You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jb...@apache.org on 2016/11/17 23:11:39 UTC

[01/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Repository: incubator-impala
Updated Branches:
  refs/heads/master 46f5ad48e -> 3be0f122a


http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_udf.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_udf.xml b/docs/topics/impala_udf.xml
new file mode 100644
index 0000000..1c0f159
--- /dev/null
+++ b/docs/topics/impala_udf.xml
@@ -0,0 +1,1933 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.2" id="udfs">
+
+  <title>Impala User-Defined Functions (UDFs)</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="UDFs"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      User-defined functions (frequently abbreviated as UDFs) let you code your own application logic for
+      processing column values during an Impala query. For example, a UDF could perform calculations using an
+      external math library, combine several column values into one, do geospatial calculations, or other kinds of
+      tests and transformations that are outside the scope of the built-in SQL operators and functions.
+    </p>
+
+    <p>
+      You can use UDFs to simplify query logic when producing reports, or to transform data in flexible ways when
+      copying from one table to another with the <codeph>INSERT ... SELECT</codeph> syntax.
+    </p>
+
+    <p>
+      You might be familiar with this feature from other database products, under names such as stored functions or
+      stored routines.
+<!--
+    , user-defined aggregate functions (UDAFs), table functions, or window functions.
+    -->
+    </p>
+
+    <p>
+      Impala support for UDFs is available in Impala 1.2 and higher:
+    </p>
+
+    <ul>
+      <li>
+        In Impala 1.1, using UDFs in a query required using the Hive shell. (Because Impala and Hive share the same
+        metastore database, you could switch to Hive to run just those queries requiring UDFs, then switch back to
+        Impala.)
+      </li>
+
+      <li>
+        Starting in Impala 1.2, Impala can run both high-performance native code UDFs written in C++, and
+        Java-based Hive UDFs that you might already have written.
+      </li>
+
+      <li>
+        Impala can run scalar UDFs that return a single value for each row of the result set, and user-defined
+        aggregate functions (UDAFs) that return a value based on a set of rows. Currently, Impala does not support
+        user-defined table functions (UDTFs) or window functions.
+      </li>
+    </ul>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="udf_concepts">
+
+    <title>UDF Concepts</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Concepts"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        Depending on your use case, you might write all-new functions, reuse Java UDFs that you have already
+        written for Hive, or port Hive Java UDF code to higher-performance native Impala UDFs in C++. You can code
+        either scalar functions for producing results one row at a time, or more complex aggregate functions for
+        doing analysis across. The following sections discuss these different aspects of working with UDFs.
+      </p>
+
+      <p outputclass="toc inpage"/>
+    </conbody>
+
+    <concept id="udfs_udafs">
+
+      <title>UDFs and UDAFs</title>
+
+      <conbody>
+
+        <p>
+          Depending on your use case, the user-defined functions (UDFs) you write might accept or produce different
+          numbers of input and output values:
+        </p>
+
+        <ul>
+          <li>
+            The most general kind of user-defined function (the one typically referred to by the abbreviation UDF)
+            takes a single input value and produces a single output value. When used in a query, it is called once
+            for each row in the result set. For example:
+<codeblock>select customer_name, is_frequent_customer(customer_id) from customers;
+select obfuscate(sensitive_column) from sensitive_data;</codeblock>
+          </li>
+
+          <li>
+            A user-defined aggregate function (UDAF) accepts a group of values and returns a single value. You use
+            UDAFs to summarize and condense sets of rows, in the same style as the built-in <codeph>COUNT</codeph>,
+            <codeph>MAX()</codeph>, <codeph>SUM()</codeph>, and <codeph>AVG()</codeph> functions. When called in a
+            query that uses the <codeph>GROUP BY</codeph> clause, the function is called once for each combination
+            of <codeph>GROUP BY</codeph> values. For example:
+<codeblock>-- Evaluates multiple rows but returns a single value.
+select closest_restaurant(latitude, longitude) from places;
+
+-- Evaluates batches of rows and returns a separate value for each batch.
+select most_profitable_location(store_id, sales, expenses, tax_rate, depreciation) from franchise_data group by year;</codeblock>
+          </li>
+
+          <li>
+            Currently, Impala does not support other categories of user-defined functions, such as user-defined
+            table functions (UDTFs) or window functions.
+          </li>
+
+<!--
+<li>
+A user-defined table function (UDTF) returns an arbitrary number of rows (zero, one, or many) for each input row.
+These functions filter, explode, or transform the input data in a variety of ways.
+Currently, Impala does not support UDTFs.
+For example:
+<codeblock>select anomalous_event() from web_traffic;
+select price_change() from stock_ticker;
+select real_words(letters) from word_games;</codeblock>
+</li>
+-->
+        </ul>
+      </conbody>
+    </concept>
+
+    <concept id="native_udfs">
+
+      <title>Native Impala UDFs</title>
+
+      <conbody>
+
+        <p>
+          Impala supports UDFs written in C++, in addition to supporting existing Hive UDFs written in Java.
+          <ph rev="upstream">Cloudera</ph> recommends using C++ UDFs because the compiled native code can yield higher performance, with
+          UDF execution time often 10x faster for a C++ UDF than the equivalent Java UDF.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept id="udfs_hive">
+
+      <title>Using Hive UDFs with Impala</title>
+
+      <conbody>
+
+        <p>
+          Impala can run Java-based user-defined functions (UDFs), originally written for Hive, with no changes,
+          subject to the following conditions:
+        </p>
+
+        <ul>
+          <li>
+            The parameters and return value must all use scalar data types supported by Impala. For example, complex or nested
+            types are not supported.
+          </li>
+
+          <li>
+            Currently, Hive UDFs that accept or return the <codeph>TIMESTAMP</codeph> type are not supported.
+          </li>
+
+          <li>
+            Prior to <keyword keyref="impala25_full"/> the return type must be a <q>Writable</q> type such as <codeph>Text</codeph> or
+            <codeph>IntWritable</codeph>, rather than a Java primitive type such as <codeph>String</codeph> or
+            <codeph>int</codeph>. Otherwise, the UDF returns <codeph>NULL</codeph>.
+            <ph rev="2.5.0">In <keyword keyref="impala25_full"/> and higher, this restriction is lifted, and both
+            UDF arguments and return values can be Java primitive types.</ph>
+          </li>
+
+          <li>
+            Hive UDAFs and UDTFs are not supported.
+          </li>
+
+          <li>
+            Typically, a Java UDF will execute several times slower in Impala than the equivalent native UDF
+            written in C++.
+          </li>
+          <li rev="2.5.0 IMPALA-2843 CDH-39148">
+            In <keyword keyref="impala25_full"/> and higher, you can transparently call Hive Java UDFs through Impala,
+            or call Impala Java UDFs through Hive. This feature does not apply to built-in Hive functions.
+            Any Impala Java UDFs created with older versions must be re-created using new <codeph>CREATE FUNCTION</codeph>
+            syntax, without any signature for arguments or the return value.
+          </li>
+        </ul>
+
+        <p>
+          To take full advantage of the Impala architecture and performance features, you can also write
+          Impala-specific UDFs in C++.
+        </p>
+
+        <p>
+          For background about Java-based Hive UDFs, see the
+          <xref href="https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF" scope="external" format="html">Hive
+          documentation for UDFs</xref>. For examples or tutorials for writing such UDFs, search the web for
+          related blog posts.
+        </p>
+
+        <p>
+          The ideal way to understand how to reuse Java-based UDFs (originally written for Hive) with Impala is to
+          take some of the Hive built-in functions (implemented as Java UDFs) and take the applicable JAR files
+          through the UDF deployment process for Impala, creating new UDFs with different names:
+        </p>
+
+        <ol>
+          <li>
+            Take a copy of the Hive JAR file containing the Hive built-in functions. For example, the path might be
+            like <filepath>/usr/lib/hive/lib/hive-exec-0.10.0-cdh4.2.0.jar</filepath>, with different version
+            numbers corresponding to your specific level of CDH.
+          </li>
+
+          <li>
+            Use <codeph>jar tf <varname>jar_file</varname></codeph> to see a list of the classes inside the JAR.
+            You will see names like <codeph>org/apache/hadoop/hive/ql/udf/UDFLower.class</codeph> and
+            <codeph>org/apache/hadoop/hive/ql/udf/UDFOPNegative.class</codeph>. Make a note of the names of the
+            functions you want to experiment with. When you specify the entry points for the Impala <codeph>CREATE
+            FUNCTION</codeph> statement, change the slash characters to dots and strip off the
+            <codeph>.class</codeph> suffix, for example <codeph>org.apache.hadoop.hive.ql.udf.UDFLower</codeph> and
+            <codeph>org.apache.hadoop.hive.ql.udf.UDFOPNegative</codeph>.
+          </li>
+
+          <li>
+            Copy that file to an HDFS location that Impala can read. (In the examples here, we renamed the file to
+            <filepath>hive-builtins.jar</filepath> in HDFS for simplicity.)
+          </li>
+
+          <li>
+            For each Java-based UDF that you want to call through Impala, issue a <codeph>CREATE FUNCTION</codeph>
+            statement, with a <codeph>LOCATION</codeph> clause containing the full HDFS path of the JAR file, and a
+            <codeph>SYMBOL</codeph> clause with the fully qualified name of the class, using dots as separators and
+            without the <codeph>.class</codeph> extension. Remember that user-defined functions are associated with
+            a particular database, so issue a <codeph>USE</codeph> statement for the appropriate database first, or
+            specify the SQL function name as
+            <codeph><varname>db_name</varname>.<varname>function_name</varname></codeph>. Use completely new names
+            for the SQL functions, because Impala UDFs cannot have the same name as Impala built-in functions.
+          </li>
+
+          <li>
+            Call the function from your queries, passing arguments of the correct type to match the function
+            signature. These arguments could be references to columns, arithmetic or other kinds of expressions,
+            the results of <codeph>CAST</codeph> functions to ensure correct data types, and so on.
+          </li>
+        </ol>
+
+        <example>
+
+          <title>Java UDF Example: Reusing lower() Function</title>
+
+          <p>
+            For example, the following <cmdname>impala-shell</cmdname> session creates an Impala UDF
+            <codeph>my_lower()</codeph> that reuses the Java code for the Hive <codeph>lower()</codeph>: built-in
+            function. We cannot call it <codeph>lower()</codeph> because Impala does not allow UDFs to have the
+            same name as built-in functions. From SQL, we call the function in a basic way (in a query with no
+            <codeph>WHERE</codeph> clause), directly on a column, and on the results of a string expression:
+          </p>
+
+<!-- To do: adapt for signatureless syntax per CDH-39148 / IMPALA-2843. -->
+<codeblock>[localhost:21000] &gt; create database udfs;
+[localhost:21000] &gt; use udfs;
+localhost:21000] &gt; create function lower(string) returns string location '/user/hive/udfs/hive.jar' symbol='org.apache.hadoop.hive.ql.udf.UDFLower';
+ERROR: AnalysisException: Function cannot have the same name as a builtin: lower
+[localhost:21000] &gt; create function my_lower(string) returns string location '/user/hive/udfs/hive.jar' symbol='org.apache.hadoop.hive.ql.udf.UDFLower';
+[localhost:21000] &gt; select my_lower('Some String NOT ALREADY LOWERCASE');
++----------------------------------------------------+
+| udfs.my_lower('some string not already lowercase') |
++----------------------------------------------------+
+| some string not already lowercase                  |
++----------------------------------------------------+
+Returned 1 row(s) in 0.11s
+[localhost:21000] &gt; create table t2 (s string);
+[localhost:21000] &gt; insert into t2 values ('lower'),('UPPER'),('Init cap'),('CamelCase');
+Inserted 4 rows in 2.28s
+[localhost:21000] &gt; select * from t2;
++-----------+
+| s         |
++-----------+
+| lower     |
+| UPPER     |
+| Init cap  |
+| CamelCase |
++-----------+
+Returned 4 row(s) in 0.47s
+[localhost:21000] &gt; select my_lower(s) from t2;
++------------------+
+| udfs.my_lower(s) |
++------------------+
+| lower            |
+| upper            |
+| init cap         |
+| camelcase        |
++------------------+
+Returned 4 row(s) in 0.54s
+[localhost:21000] &gt; select my_lower(concat('ABC ',s,' XYZ')) from t2;
++------------------------------------------+
+| udfs.my_lower(concat('abc ', s, ' xyz')) |
++------------------------------------------+
+| abc lower xyz                            |
+| abc upper xyz                            |
+| abc init cap xyz                         |
+| abc camelcase xyz                        |
++------------------------------------------+
+Returned 4 row(s) in 0.22s</codeblock>
+
+        </example>
+
+        <example>
+
+          <title>Java UDF Example: Reusing negative() Function</title>
+
+          <p>
+            Here is an example that reuses the Hive Java code for the <codeph>negative()</codeph> built-in
+            function. This example demonstrates how the data types of the arguments must match precisely with the
+            function signature. At first, we create an Impala SQL function that can only accept an integer
+            argument. Impala cannot find a matching function when the query passes a floating-point argument,
+            although we can call the integer version of the function by casting the argument. Then we overload the
+            same function name to also accept a floating-point argument.
+          </p>
+
+<codeblock>[localhost:21000] &gt; create table t (x int);
+[localhost:21000] &gt; insert into t values (1), (2), (4), (100);
+Inserted 4 rows in 1.43s
+[localhost:21000] &gt; create function my_neg(bigint) returns bigint location '/user/hive/udfs/hive.jar' symbol='org.apache.hadoop.hive.ql.udf.UDFOPNegative';
+[localhost:21000] &gt; select my_neg(4);
++----------------+
+| udfs.my_neg(4) |
++----------------+
+| -4             |
++----------------+
+[localhost:21000] &gt; select my_neg(x) from t;
++----------------+
+| udfs.my_neg(x) |
++----------------+
+| -2             |
+| -4             |
+| -100           |
++----------------+
+Returned 3 row(s) in 0.60s
+[localhost:21000] &gt; select my_neg(4.0);
+ERROR: AnalysisException: No matching function with signature: udfs.my_neg(FLOAT).
+[localhost:21000] &gt; select my_neg(cast(4.0 as int));
++-------------------------------+
+| udfs.my_neg(cast(4.0 as int)) |
++-------------------------------+
+| -4                            |
++-------------------------------+
+Returned 1 row(s) in 0.11s
+[localhost:21000] &gt; create function my_neg(double) returns double location '/user/hive/udfs/hive.jar' symbol='org.apache.hadoop.hive.ql.udf.UDFOPNegative';
+[localhost:21000] &gt; select my_neg(4.0);
++------------------+
+| udfs.my_neg(4.0) |
++------------------+
+| -4               |
++------------------+
+Returned 1 row(s) in 0.11s</codeblock>
+
+          <p>
+            You can find the sample files mentioned here in
+            <xref href="https://github.com/cloudera/impala/tree/master/be/src/udf_samples" scope="external" format="html">the
+            Impala github repo</xref>.
+<!-- Internal-only repo, don't know an external equivalent.
+and other examples demonstrating this technique in
+<xref href="http://github.sf.cloudera.com/CDH/Impala/blob/master/testdata/workloads/functional-query/queries/QueryTest/load-hive-udfs.test" scope="external" format="html">the Impala test files</xref>.
+-->
+          </p>
+
+        </example>
+      </conbody>
+    </concept>
+  </concept>
+
+  <concept id="udf_runtime">
+    <title>Runtime Environment for UDFs</title>
+    <conbody>
+      <p>
+        By default, Impala copies UDFs into <filepath>/tmp</filepath>,
+        and you can configure this location through the <codeph>--local_library_dir</codeph>
+        startup flag for the <cmdname>impalad</cmdname> daemon.
+      </p>
+    </conbody>
+  </concept>
+
+
+  <concept id="udf_demo_env">
+
+    <title>Installing the UDF Development Package</title>
+
+    <conbody>
+
+      <p rev="CDH-37080">
+        To develop UDFs for Impala, download and install the <codeph>impala-udf-devel</codeph> package (RHEL-based
+        distributions) or <codeph>impala-udf-dev</codeph> (Ubuntu and Debian). This package contains
+        header files, sample source, and build configuration files.
+      </p>
+
+      <ol>
+        <li>
+          Start at <xref href="https://archive.cloudera.com/cdh5/" scope="external" format="html"/> for the CDH 5
+          package, or <xref href="https://archive.cloudera.com/impala/" scope="external" format="html"/> for the CDH
+          4 package.
+        </li>
+
+        <li>
+          Locate the appropriate <codeph>.repo</codeph> or list file for your operating system version, such as
+          <xref href="https://archive.cloudera.com/impala/redhat/6/x86_64/impala/cloudera-impala.repo" scope="external" format="html">the
+          <codeph>.repo</codeph> file for CDH 4 on RHEL 6</xref>.
+        </li>
+
+        <li rev="CDH-37080">
+          Use the familiar <codeph>yum</codeph>, <codeph>zypper</codeph>, or <codeph>apt-get</codeph> commands
+          depending on your operating system. For the package name, specify <codeph>impala-udf-devel</codeph>
+          (RHEL-based distributions) or <codeph>impala-udf-dev</codeph> (Ubuntu and Debian).
+        </li>
+      </ol>
+
+      <note>
+        The UDF development code does not rely on Impala being installed on the same machine. You can write and
+        compile UDFs on a minimal development system, then deploy them on a different one for use with Impala. If
+        you develop UDFs on a server managed by Cloudera Manager through the parcel mechanism, you still install
+        the UDF development kit through the package mechanism; this small standalone package does not interfere
+        with the parcels containing the main Impala code.
+      </note>
+
+      <p>
+        When you are ready to start writing your own UDFs, download the sample code and build scripts from
+        <xref href="https://github.com/cloudera/impala-udf-samples" scope="external" format="html">the Cloudera
+        sample UDF github</xref>. Then see <xref href="impala_udf.xml#udf_coding"/> for how to code UDFs, and
+        <xref href="impala_udf.xml#udf_tutorial"/> for how to build and run UDFs.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="udf_coding">
+
+    <title>Writing User-Defined Functions (UDFs)</title>
+
+    <conbody>
+
+      <p>
+        Before starting UDF development, make sure to install the development package and download the UDF code
+        samples, as described in <xref href="#udf_demo_env"/>.
+      </p>
+
+      <p>
+        When writing UDFs:
+      </p>
+
+      <ul>
+        <li>
+          Keep in mind the data type differences as you transfer values from the high-level SQL to your lower-level
+          UDF code. For example, in the UDF code you might be much more aware of how many bytes different kinds of
+          integers require.
+        </li>
+
+        <li>
+          Use best practices for function-oriented programming: choose arguments carefully, avoid side effects,
+          make each function do a single thing, and so on.
+        </li>
+      </ul>
+
+      <p outputclass="toc inpage"/>
+    </conbody>
+
+    <concept id="udf_exploring">
+
+      <title>Getting Started with UDF Coding</title>
+  <prolog>
+    <metadata>
+      <!-- OK, this is not something a Hadoop newbie would tackle, but being lenient and inclusive in this initial pass, so including the GS tag. -->
+      <data name="Category" value="Getting Started"/>
+    </metadata>
+  </prolog>
+
+      <conbody>
+
+        <p>
+          To understand the layout and member variables and functions of the predefined UDF data types, examine the
+          header file <filepath>/usr/include/impala_udf/udf.h</filepath>:
+        </p>
+
+<codeblock>// This is the only Impala header required to develop UDFs and UDAs. This header
+// contains the types that need to be used and the FunctionContext object. The context
+// object serves as the interface object between the UDF/UDA and the impala process. </codeblock>
+
+        <p>
+          For the basic declarations needed to write a scalar UDF, see the header file
+          <xref href="https://github.com/cloudera/impala-udf-samples/blob/master/udf-sample.h" scope="external" format="html"><filepath>udf-sample.h</filepath></xref>
+          within the sample build environment, which defines a simple function
+          named <codeph>AddUdf()</codeph>:
+        </p>
+
+<!-- Downloadable version of this file: https://raw.githubusercontent.com/cloudera/impala-udf-samples/master/udf-sample.h -->
+<codeblock>#ifndef IMPALA_UDF_SAMPLE_UDF_H
+#define IMPALA_UDF_SAMPLE_UDF_H
+
+#include &lt;impala_udf/udf.h&gt;
+
+using namespace impala_udf;
+
+IntVal AddUdf(FunctionContext* context, const IntVal&amp; arg1, const IntVal&amp; arg2);
+
+#endif
+</codeblock>
+
+        <p>
+          For sample C++ code for a simple function named <codeph>AddUdf()</codeph>, see the source file
+          <filepath>udf-sample.cc</filepath> within the sample build environment:
+        </p>
+
+<!-- Downloadable version of this file: https://raw.githubusercontent.com/cloudera/impala-udf-samples/master/udf-sample.cc -->
+<codeblock>#include "udf-sample.h"
+
+// In this sample we are declaring a UDF that adds two ints and returns an int.
+IntVal AddUdf(FunctionContext* context, const IntVal&amp; arg1, const IntVal&amp; arg2) {
+  if (arg1.is_null || arg2.is_null) return IntVal::null();
+  return IntVal(arg1.val + arg2.val);
+}
+
+// Multiple UDFs can be defined in the same file</codeblock>
+      </conbody>
+    </concept>
+
+    <concept id="udfs_args">
+
+      <title>Data Types for Function Arguments and Return Values</title>
+
+      <conbody>
+
+        <p>
+          Each value that a user-defined function can accept as an argument or return as a result value must map to
+          a SQL data type that you could specify for a table column.
+        </p>
+
+        <p conref="../shared/impala_common.xml#common/udfs_no_complex_types"/>
+
+        <p>
+          Each data type has a corresponding structure defined in the C++ and Java header files, with two member
+          fields and some predefined comparison operators and constructors:
+        </p>
+
+        <ul>
+          <li>
+            <p>
+              <codeph>is_null</codeph> indicates whether the value is <codeph>NULL</codeph> or not.
+              <codeph>val</codeph> holds the actual argument or return value when it is non-<codeph>NULL</codeph>.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              Each struct also defines a <codeph>null()</codeph> member function that constructs an instance of the
+              struct with the <codeph>is_null</codeph> flag set.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              The built-in SQL comparison operators and clauses such as <codeph>&lt;</codeph>,
+              <codeph>&gt;=</codeph>, <codeph>BETWEEN</codeph>, and <codeph>ORDER BY</codeph> all work
+              automatically based on the SQL return type of each UDF. For example, Impala knows how to evaluate
+              <codeph>BETWEEN 1 AND udf_returning_int(col1)</codeph> or <codeph>ORDER BY
+              udf_returning_string(col2)</codeph> without you declaring any comparison operators within the UDF
+              itself.
+            </p>
+            <p>
+              For convenience within your UDF code, each struct defines <codeph>==</codeph> and <codeph>!=</codeph>
+              operators for comparisons with other structs of the same type. These are for typical C++ comparisons
+              within your own code, not necessarily reproducing SQL semantics. For example, if the
+              <codeph>is_null</codeph> flag is set in both structs, they compare as equal. That behavior of
+              <codeph>null</codeph> comparisons is different from SQL (where <codeph>NULL == NULL</codeph> is
+              <codeph>NULL</codeph> rather than <codeph>true</codeph>), but more in line with typical C++ behavior.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              Each kind of struct has one or more constructors that define a filled-in instance of the struct,
+              optionally with default values.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              Each kind of struct has a <codeph>null()</codeph> member function that returns an instance of the
+              struct with the <codeph>is_null</codeph> flag set.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              Because Impala currently does not support composite or nested types, Impala cannot process UDFs that
+              accept such types as arguments or return them as result values. This limitation applies both to
+              Impala UDFs written in C++ and Java-based Hive UDFs.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              You can overload functions by creating multiple functions with the same SQL name but different
+              argument types. For overloaded functions, you must use different C++ or Java entry point names in the
+              underlying functions.
+            </p>
+          </li>
+        </ul>
+
+        <p>
+          The data types defined on the C++ side (in <filepath>/usr/include/impala_udf/udf.h</filepath>) are:
+        </p>
+
+        <ul>
+          <li>
+            <p>
+              <codeph>IntVal</codeph> represents an <codeph>INT</codeph> column.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              <codeph>BigIntVal</codeph> represents a <codeph>BIGINT</codeph> column. Even if you do not need the
+              full range of a <codeph>BIGINT</codeph> value, it can be useful to code your function arguments as
+              <codeph>BigIntVal</codeph> to make it convenient to call the function with different kinds of integer
+              columns and expressions as arguments. Impala automatically casts smaller integer types to larger ones
+              when appropriate, but does not implicitly cast large integer types to smaller ones.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              <codeph>SmallIntVal</codeph> represents a <codeph>SMALLINT</codeph> column.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              <codeph>TinyIntVal</codeph> represents a <codeph>TINYINT</codeph> column.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              <codeph>StringVal</codeph> represents a <codeph>STRING</codeph> column. It has a <codeph>len</codeph>
+              field representing the length of the string, and a <codeph>ptr</codeph> field pointing to the string
+              data. It has constructors that create a new <codeph>StringVal</codeph> struct based on a
+              null-terminated C-style string, or a pointer plus a length; these new structs still refer to the
+              original string data rather than allocating a new buffer for the data. It also has a constructor that
+              takes a pointer to a <codeph>FunctionContext</codeph> struct and a length, that does allocate space
+              for a new copy of the string data, for use in UDFs that return string values.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              <codeph>BooleanVal</codeph> represents a <codeph>BOOLEAN</codeph> column.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              <codeph>FloatVal</codeph> represents a <codeph>FLOAT</codeph> column.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              <codeph>DoubleVal</codeph> represents a <codeph>DOUBLE</codeph> column.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              <codeph>TimestampVal</codeph> represents a <codeph>TIMESTAMP</codeph> column. It has a
+              <codeph>date</codeph> field, a 32-bit integer representing the Gregorian date, that is, the days past
+              the epoch date. It also has a <codeph>time_of_day</codeph> field, a 64-bit integer representing the
+              current time of day in nanoseconds.
+            </p>
+          </li>
+
+<!--
+  <li>
+    <p>
+      <codeph>AnyVal</codeph> is the parent type of all the other
+      structs. They inherit the <codeph>is_null</codeph> field from it.
+      You do not use this type directly in your code.
+    </p>
+  </li>
+-->
+        </ul>
+      </conbody>
+    </concept>
+
+    <concept id="udf_varargs">
+
+      <title>Variable-Length Argument Lists</title>
+
+      <conbody>
+
+        <p>
+          UDFs typically take a fixed number of arguments, with each one named explicitly in the signature of your
+          C++ function. Your function can also accept additional optional arguments, all of the same type. For
+          example, you can concatenate two strings, three strings, four strings, and so on. Or you can compare two
+          numbers, three numbers, four numbers, and so on.
+        </p>
+
+        <p>
+          To accept a variable-length argument list, code the signature of your function like this:
+        </p>
+
+<codeblock>StringVal Concat(FunctionContext* context, const StringVal&amp; separator,
+  int num_var_args, const StringVal* args);</codeblock>
+
+        <p>
+          In the <codeph>CREATE FUNCTION</codeph> statement, after the type of the first optional argument, include
+          <codeph>...</codeph> to indicate it could be followed by more arguments of the same type. For example,
+          the following function accepts a <codeph>STRING</codeph> argument, followed by one or more additional
+          <codeph>STRING</codeph> arguments:
+        </p>
+
+<codeblock>[localhost:21000] &gt; create function my_concat(string, string ...) returns string location '/user/test_user/udfs/sample.so' symbol='Concat';
+</codeblock>
+
+        <p>
+          The call from the SQL query must pass at least one argument to the variable-length portion of the
+          argument list.
+        </p>
+
+        <p>
+          When Impala calls the function, it fills in the initial set of required arguments, then passes the number
+          of extra arguments and a pointer to the first of those optional arguments.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept id="udf_null">
+
+      <title>Handling NULL Values</title>
+
+      <conbody>
+
+        <p>
+          For correctness, performance, and reliability, it is important for each UDF to handle all situations
+          where any <codeph>NULL</codeph> values are passed to your function. For example, when passed a
+          <codeph>NULL</codeph>, UDFs typically also return <codeph>NULL</codeph>. In an aggregate function, which
+          could be passed a combination of real and <codeph>NULL</codeph> values, you might make the final value
+          into a <codeph>NULL</codeph> (as in <codeph>CONCAT()</codeph>), ignore the <codeph>NULL</codeph> value
+          (as in <codeph>AVG()</codeph>), or treat it the same as a numeric zero or empty string.
+        </p>
+
+        <p>
+          Each parameter type, such as <codeph>IntVal</codeph> or <codeph>StringVal</codeph>, has an
+          <codeph>is_null</codeph> Boolean member.
+<!--
+If your function has no effect when passed <codeph>NULL</codeph>
+values,
+-->
+          Test this flag immediately for each argument to your function, and if it is set, do not refer to the
+          <codeph>val</codeph> field of the argument structure. The <codeph>val</codeph> field is undefined when
+          the argument is <codeph>NULL</codeph>, so your function could go into an infinite loop or produce
+          incorrect results if you skip the special handling for <codeph>NULL</codeph>.
+<!-- and return if so.
+For <codeph>void</codeph> intermediate functions
+within UDAs, you can return without specifying a value.
+-->
+        </p>
+
+        <p>
+          If your function returns <codeph>NULL</codeph> when passed a <codeph>NULL</codeph> value, or in other
+          cases such as when a search string is not found, you can construct a null instance of the return type by
+          using its <codeph>null()</codeph> member function.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept id="udf_malloc">
+
+      <title>Memory Allocation for UDFs</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Memory"/>
+    </metadata>
+  </prolog>
+
+      <conbody>
+
+        <p>
+          By default, memory allocated within a UDF is deallocated when the function exits, which could be before
+          the query is finished. The input arguments remain allocated for the lifetime of the function, so you can
+          refer to them in the expressions for your return values. If you use temporary variables to construct
+          all-new string values, use the <codeph>StringVal()</codeph> constructor that takes an initial
+          <codeph>FunctionContext*</codeph> argument followed by a length, and copy the data into the newly
+          allocated memory buffer.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept rev="1.3.0" id="udf_threads">
+
+      <title>Thread-Safe Work Area for UDFs</title>
+
+      <conbody>
+
+        <p>
+          One way to improve performance of UDFs is to specify the optional <codeph>PREPARE_FN</codeph> and
+          <codeph>CLOSE_FN</codeph> clauses on the <codeph>CREATE FUNCTION</codeph> statement. The <q>prepare</q>
+          function sets up a thread-safe data structure in memory that you can use as a work area. The <q>close</q>
+          function deallocates that memory. Each subsequent call to the UDF within the same thread can access that
+          same memory area. There might be several such memory areas allocated on the same host, as UDFs are
+          parallelized using multiple threads.
+        </p>
+
+        <p>
+          Within this work area, you can set up predefined lookup tables, or record the results of complex
+          operations on data types such as <codeph>STRING</codeph> or <codeph>TIMESTAMP</codeph>. Saving the
+          results of previous computations rather than repeating the computation each time is an optimization known
+          as <xref href="http://en.wikipedia.org/wiki/Memoization" scope="external" format="html"/>. For example,
+          if your UDF performs a regular expression match or date manipulation on a column that repeats the same
+          value over and over, you could store the last-computed value or a hash table of already-computed values,
+          and do a fast lookup to find the result for subsequent iterations of the UDF.
+        </p>
+
+        <p>
+          Each such function must have the signature:
+        </p>
+
+<codeblock>void <varname>function_name</varname>(impala_udf::FunctionContext*, impala_udf::FunctionContext::FunctionScope)
+</codeblock>
+
+        <p>
+          Currently, only <codeph>THREAD_SCOPE</codeph> is implemented, not <codeph>FRAGMENT_SCOPE</codeph>. See
+          <filepath>udf.h</filepath> for details about the scope values.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept id="udf_error_handling">
+
+      <title>Error Handling for UDFs</title>
+  <prolog>
+    <metadata>
+      <!-- A little bit of a stretch, but if you're doing UDFs and you need to debug you might look up Troubleshooting. -->
+      <data name="Category" value="Troubleshooting"/>
+    </metadata>
+  </prolog>
+
+      <conbody>
+
+        <p>
+          To handle errors in UDFs, you call functions that are members of the initial
+          <codeph>FunctionContext*</codeph> argument passed to your function.
+        </p>
+
+        <p>
+          A UDF can record one or more warnings, for conditions that indicate minor, recoverable problems that do
+          not cause the query to stop. The signature for this function is:
+        </p>
+
+<codeblock>bool AddWarning(const char* warning_msg);</codeblock>
+
+        <p>
+          For a serious problem that requires cancelling the query, a UDF can set an error flag that prevents the
+          query from returning any results. The signature for this function is:
+        </p>
+
+<codeblock>void SetError(const char* error_msg);</codeblock>
+      </conbody>
+    </concept>
+  </concept>
+
+  <concept id="udafs">
+
+    <title>Writing User-Defined Aggregate Functions (UDAFs)</title>
+
+    <conbody>
+
+      <p>
+        User-defined aggregate functions (UDAFs or UDAs) are a powerful and flexible category of user-defined
+        functions. If a query processes N rows, calling a UDAF during the query condenses the result set, anywhere
+        from a single value (such as with the <codeph>SUM</codeph> or <codeph>MAX</codeph> functions), or some
+        number less than or equal to N (as in queries using the <codeph>GROUP BY</codeph> or
+        <codeph>HAVING</codeph> clause).
+      </p>
+
+      <p outputclass="toc inpage"/>
+    </conbody>
+
+    <concept id="uda_functions">
+
+      <title>The Underlying Functions for a UDA</title>
+
+      <conbody>
+
+        <p>
+          A UDAF must maintain a state value across subsequent calls, so that it can accumulate a result across a
+          set of calls, rather than derive it purely from one set of arguments. For that reason, a UDAF is
+          represented by multiple underlying functions:
+        </p>
+
+        <ul>
+          <li>
+            An initialization function that sets any counters to zero, creates empty buffers, and does any other
+            one-time setup for a query.
+          </li>
+
+          <li>
+            An update function that processes the arguments for each row in the query result set and accumulates an
+            intermediate result for each node. For example, this function might increment a counter, append to a
+            string buffer, or set flags.
+          </li>
+
+          <li>
+            A merge function that combines the intermediate results from two different nodes.
+          </li>
+
+          <li rev="2.0.0">
+            A serialize function that flattens any intermediate values containing pointers, and frees any memory
+            allocated during the init, update, and merge phases.
+          </li>
+
+          <li>
+            A finalize function that either passes through the combined result unchanged, or does one final
+            transformation.
+          </li>
+        </ul>
+
+        <p>
+          In the SQL syntax, you create a UDAF by using the statement <codeph>CREATE AGGREGATE FUNCTION</codeph>.
+          You specify the entry points of the underlying C++ functions using the clauses <codeph>INIT_FN</codeph>,
+          <codeph>UPDATE_FN</codeph>, <codeph>MERGE_FN</codeph>, <codeph rev="2.0.0">SERIALIZE_FN</codeph>, and
+          <codeph>FINALIZE_FN</codeph>.
+        </p>
+
+        <p>
+          <!-- To do:
+            Need an example to demonstrate exactly what tokens are used for init, merge, finalize in
+            this substitution.
+          -->
+          For convenience, you can use a naming convention for the underlying functions and Impala automatically
+          recognizes those entry points. Specify the <codeph>UPDATE_FN</codeph> clause, using an entry point name
+          containing the string <codeph>update</codeph> or <codeph>Update</codeph>. When you omit the other
+          <codeph>_FN</codeph> clauses from the SQL statement, Impala looks for entry points with names formed by
+          substituting the <codeph>update</codeph> or <codeph>Update</codeph> portion of the specified name.
+        </p>
+
+<!--
+[INIT_FN '<varname>function</varname>]
+[UPDATE_FN '<varname>function</varname>]
+[MERGE_FN '<varname>function</varname>]
+[FINALIZE_FN '<varname>function</varname>]
+-->
+
+        <p>
+          <filepath>uda-sample.h</filepath>:
+        </p>
+
+        <p> See this file online at: <xref
+            href="https://github.com/cloudera/impala-udf-samples/blob/master/uda-sample.h"
+            scope="external" format="html" /></p>
+
+<codeblock audience="Cloudera">#ifndef SAMPLES_UDA_H
+#define SAMPLES_UDA_H
+
+#include &lt;impala_udf/udf.h&gt;
+
+using namespace impala_udf;
+
+// This is an example of the COUNT aggregate function.
+//
+// Usage: &gt; create aggregate function my_count(int) returns bigint
+//          location '/user/cloudera/libudasample.so' update_fn='CountUpdate';
+//        &gt; select my_count(col) from tbl;
+
+void CountInit(FunctionContext* context, BigIntVal* val);
+void CountUpdate(FunctionContext* context, const IntVal&amp; input, BigIntVal* val);
+void CountMerge(FunctionContext* context, const BigIntVal&amp; src, BigIntVal* dst);
+BigIntVal CountFinalize(FunctionContext* context, const BigIntVal&amp; val);
+
+
+// This is an example of the AVG(double) aggregate function. This function needs to
+// maintain two pieces of state, the current sum and the count. We do this using
+// the StringVal intermediate type. When this UDA is registered, it would specify
+// 16 bytes (8 byte sum + 8 byte count) as the size for this buffer.
+//
+// Usage: &gt; create aggregate function my_avg(double) returns string 
+//          location '/user/cloudera/libudasample.so' update_fn='AvgUpdate';
+//        &gt; select cast(my_avg(col) as double) from tbl;
+
+void AvgInit(FunctionContext* context, StringVal* val);
+void AvgUpdate(FunctionContext* context, const DoubleVal&amp; input, StringVal* val);
+void AvgMerge(FunctionContext* context, const StringVal&amp; src, StringVal* dst);
+const StringVal AvgSerialize(FunctionContext* context, const StringVal&amp; val);
+StringVal AvgFinalize(FunctionContext* context, const StringVal&amp; val);
+
+
+// This is a sample of implementing the STRING_CONCAT aggregate function.
+//
+// Usage: &gt; create aggregate function string_concat(string, string) returns string
+//          location '/user/cloudera/libudasample.so' update_fn='StringConcatUpdate';
+//        &gt; select string_concat(string_col, ",") from table;
+
+void StringConcatInit(FunctionContext* context, StringVal* val);
+void StringConcatUpdate(FunctionContext* context, const StringVal&amp; arg1,
+    const StringVal&amp; arg2, StringVal* val);
+void StringConcatMerge(FunctionContext* context, const StringVal&amp; src, StringVal* dst);
+const StringVal StringConcatSerialize(FunctionContext* context, const StringVal&amp; val);
+StringVal StringConcatFinalize(FunctionContext* context, const StringVal&amp; val);
+
+
+// This is a example of the variance aggregate function.
+//
+// Usage: &gt; create aggregate function var(double) returns string
+//          location '/user/cloudera/libudasample.so' update_fn='VarianceUpdate';
+//        &gt; select cast(var(col) as double) from tbl;
+
+void VarianceInit(FunctionContext* context, StringVal* val);
+void VarianceUpdate(FunctionContext* context, const DoubleVal&amp; input, StringVal* val);
+void VarianceMerge(FunctionContext* context, const StringVal&amp; src, StringVal* dst);
+const StringVal VarianceSerialize(FunctionContext* context, const StringVal&amp; val);
+StringVal VarianceFinalize(FunctionContext* context, const StringVal&amp; val);
+
+
+// An implementation of the Knuth online variance algorithm, which is also single pass and
+// more numerically stable.
+//
+// Usage: &gt; create aggregate function knuth_var(double) returns string
+//          location '/user/cloudera/libudasample.so' update_fn='KnuthVarianceUpdate';
+//        &gt; select cast(knuth_var(col) as double) from tbl;
+
+void KnuthVarianceInit(FunctionContext* context, StringVal* val);
+void KnuthVarianceUpdate(FunctionContext* context, const DoubleVal&amp; input, StringVal* val);
+void KnuthVarianceMerge(FunctionContext* context, const StringVal&amp; src, StringVal* dst);
+const StringVal KnuthVarianceSerialize(FunctionContext* context, const StringVal&amp; val);
+StringVal KnuthVarianceFinalize(FunctionContext* context, const StringVal&amp; val);
+
+
+// The different steps of the UDA are composable. In this case, we'the UDA will use the
+// other steps from the Knuth variance computation.
+//
+// Usage: &gt; create aggregate function stddev(double) returns string
+//          location '/user/cloudera/libudasample.so' update_fn='KnuthVarianceUpdate'
+//          finalize_fn="StdDevFinalize";
+//        &gt; select cast(stddev(col) as double) from tbl;
+
+StringVal StdDevFinalize(FunctionContext* context, const StringVal&amp; val);
+
+
+// Utility function for serialization to StringVal
+template &lt;typename T&gt;
+StringVal ToStringVal(FunctionContext* context, const T&amp; val);
+
+#endif</codeblock>
+
+        <p>
+          <filepath>uda-sample.cc</filepath>:
+        </p>
+
+        <p> See this file online at: <xref
+            href="https://github.com/cloudera/impala-udf-samples/blob/master/uda-sample.cc"
+            scope="external" format="html" />
+        </p>
+
+<codeblock audience="Cloudera">#include "uda-sample.h"
+#include &lt;assert.h&gt;
+#include &lt;sstream&gt;
+
+using namespace impala_udf;
+using namespace std;
+
+template &lt;typename T&gt;
+StringVal ToStringVal(FunctionContext* context, const T&amp; val) {
+  stringstream ss;
+  ss &lt;&lt; val;
+  string str = ss.str();
+  StringVal string_val(context, str.size());
+  memcpy(string_val.ptr, str.c_str(), str.size());
+  return string_val;
+}
+
+template &lt;&gt;
+StringVal ToStringVal&lt;DoubleVal&gt;(FunctionContext* context, const DoubleVal&amp; val) {
+  if (val.is_null) return StringVal::null();
+  return ToStringVal(context, val.val);
+}
+
+// ---------------------------------------------------------------------------
+// This is a sample of implementing a COUNT aggregate function.
+// ---------------------------------------------------------------------------
+void CountInit(FunctionContext* context, BigIntVal* val) {
+  val-&gt;is_null = false;
+  val-&gt;val = 0;
+}
+
+void CountUpdate(FunctionContext* context, const IntVal&amp; input, BigIntVal* val) {
+  if (input.is_null) return;
+  ++val-&gt;val;
+}
+
+void CountMerge(FunctionContext* context, const BigIntVal&amp; src, BigIntVal* dst) {
+  dst-&gt;val += src.val;
+}
+
+BigIntVal CountFinalize(FunctionContext* context, const BigIntVal&amp; val) {
+  return val;
+}
+
+// ---------------------------------------------------------------------------
+// This is a sample of implementing a AVG aggregate function.
+// ---------------------------------------------------------------------------
+struct AvgStruct {
+  double sum;
+  int64_t count;
+};
+
+// Initialize the StringVal intermediate to a zero'd AvgStruct
+void AvgInit(FunctionContext* context, StringVal* val) {
+  val-&gt;is_null = false;
+  val-&gt;len = sizeof(AvgStruct);
+  val-&gt;ptr = context-&gt;Allocate(val-&gt;len);
+  memset(val-&gt;ptr, 0, val-&gt;len);
+}
+
+void AvgUpdate(FunctionContext* context, const DoubleVal&amp; input, StringVal* val) {
+  if (input.is_null) return;
+  assert(!val-&gt;is_null);
+  assert(val-&gt;len == sizeof(AvgStruct));
+  AvgStruct* avg = reinterpret_cast&lt;AvgStruct*&gt;(val-&gt;ptr);
+  avg-&gt;sum += input.val;
+  ++avg-&gt;count;
+}
+
+void AvgMerge(FunctionContext* context, const StringVal&amp; src, StringVal* dst) {
+  if (src.is_null) return;
+  const AvgStruct* src_avg = reinterpret_cast&lt;const AvgStruct*&gt;(src.ptr);
+  AvgStruct* dst_avg = reinterpret_cast&lt;AvgStruct*&gt;(dst-&gt;ptr);
+  dst_avg-&gt;sum += src_avg-&gt;sum;
+  dst_avg-&gt;count += src_avg-&gt;count;
+}
+
+// A serialize function is necesary to free the intermediate state allocation. We use the
+// StringVal constructor to allocate memory owned by Impala, copy the intermediate state,
+// and free the original allocation. Note that memory allocated by the StringVal ctor is
+// not necessarily persisted across UDA function calls, which is why we don't use it in
+// AvgInit().
+const StringVal AvgSerialize(FunctionContext* context, const StringVal&amp; val) {
+  assert(!val.is_null);
+  StringVal result(context, val.len);
+  memcpy(result.ptr, val.ptr, val.len);
+  context-&gt;Free(val.ptr);
+  return result;
+}
+
+StringVal AvgFinalize(FunctionContext* context, const StringVal&amp; val) {
+  assert(!val.is_null);
+  assert(val.len == sizeof(AvgStruct));
+  AvgStruct* avg = reinterpret_cast&lt;AvgStruct*&gt;(val.ptr);
+  StringVal result;
+  if (avg-&gt;count == 0) {
+    result = StringVal::null();
+  } else {
+    // Copies the result to memory owned by Impala
+    result = ToStringVal(context, avg-&gt;sum / avg-&gt;count);
+  }
+  context-&gt;Free(val.ptr);
+  return result;
+}
+
+// ---------------------------------------------------------------------------
+// This is a sample of implementing the STRING_CONCAT aggregate function.
+// Example: select string_concat(string_col, ",") from table
+// ---------------------------------------------------------------------------
+// Delimiter to use if the separator is NULL.
+static const StringVal DEFAULT_STRING_CONCAT_DELIM((uint8_t*)", ", 2);
+
+void StringConcatInit(FunctionContext* context, StringVal* val) {
+  val-&gt;is_null = true;
+}
+
+void StringConcatUpdate(FunctionContext* context, const StringVal&amp; str,
+    const StringVal&amp; separator, StringVal* result) {
+  if (str.is_null) return;
+  if (result-&gt;is_null) {
+    // This is the first string, simply set the result to be the value.
+    uint8_t* copy = context-&gt;Allocate(str.len);
+    memcpy(copy, str.ptr, str.len);
+    *result = StringVal(copy, str.len);
+    return;
+  }
+
+  const StringVal* sep_ptr = separator.is_null ? &amp;DEFAULT_STRING_CONCAT_DELIM :
+      &amp;separator;
+
+  // We need to grow the result buffer and then append the new string and
+  // separator.
+  int new_size = result-&gt;len + sep_ptr-&gt;len + str.len;
+  result-&gt;ptr = context-&gt;Reallocate(result-&gt;ptr, new_size);
+  memcpy(result-&gt;ptr + result-&gt;len, sep_ptr-&gt;ptr, sep_ptr-&gt;len);
+  result-&gt;len += sep_ptr-&gt;len;
+  memcpy(result-&gt;ptr + result-&gt;len, str.ptr, str.len);
+  result-&gt;len += str.len;
+}
+
+void StringConcatMerge(FunctionContext* context, const StringVal&amp; src, StringVal* dst) {
+  if (src.is_null) return;
+  StringConcatUpdate(context, src, ",", dst);
+}
+
+// A serialize function is necesary to free the intermediate state allocation. We use the
+// StringVal constructor to allocate memory owned by Impala, copy the intermediate
+// StringVal, and free the intermediate's memory. Note that memory allocated by the
+// StringVal ctor is not necessarily persisted across UDA function calls, which is why we
+// don't use it in StringConcatUpdate().
+const StringVal StringConcatSerialize(FunctionContext* context, const StringVal&amp; val) {
+  if (val.is_null) return val;
+  StringVal result(context, val.len);
+  memcpy(result.ptr, val.ptr, val.len);
+  context-&gt;Free(val.ptr);
+  return result;
+}
+
+// Same as StringConcatSerialize().
+StringVal StringConcatFinalize(FunctionContext* context, const StringVal&amp; val) {
+  if (val.is_null) return val;
+  StringVal result(context, val.len);
+  memcpy(result.ptr, val.ptr, val.len);
+  context-&gt;Free(val.ptr);
+  return result;
+}</codeblock>
+      </conbody>
+    </concept>
+
+    <concept rev="2.3.0 IMPALA-1829 CDH-30572" id="udf_intermediate">
+
+      <title>Intermediate Results for UDAs</title>
+
+      <conbody>
+
+        <p>
+          A user-defined aggregate function might produce and combine intermediate results during some phases of
+          processing, using a different data type than the final return value. For example, if you implement a
+          function similar to the built-in <codeph>AVG()</codeph> function, it must keep track of two values, the
+          number of values counted and the sum of those values. Or, you might accumulate a string value over the
+          course of a UDA, then in the end return a numeric or Boolean result.
+        </p>
+
+        <p>
+          In such a case, specify the data type of the intermediate results using the optional <codeph>INTERMEDIATE
+          <varname>type_name</varname></codeph> clause of the <codeph>CREATE AGGREGATE FUNCTION</codeph> statement.
+          If the intermediate data is a typeless byte array (for example, to represent a C++ struct or array),
+          specify the type name as <codeph>CHAR(<varname>n</varname>)</codeph>, with <varname>n</varname>
+          representing the number of bytes in the intermediate result buffer.
+        </p>
+
+        <p>
+          For an example of this technique, see the <codeph>trunc_sum()</codeph> aggregate function, which accumulates
+          intermediate results of type <codeph>DOUBLE</codeph> and returns <codeph>BIGINT</codeph> at the end.
+          View
+          <xref href="https://github.com/cloudera/Impala/blob/cdh5-trunk/tests/query_test/test_udfs.py" scope="external" format="html">the <codeph>CREATE FUNCTION</codeph> statement</xref>
+          and
+          <xref href="http://github.com/Cloudera/Impala/blob/cdh5-trunk/be/src/testutil/test-udas.cc" scope="external" format="html">the implementation of the underlying TruncSum*() functions</xref>
+          on Github.
+        </p>
+      </conbody>
+    </concept>
+  </concept>
+
+  <concept id="udf_building">
+
+    <title>Building and Deploying UDFs</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Deploying"/>
+      <data name="Category" value="Building"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        This section explains the steps to compile Impala UDFs from C++ source code, and deploy the resulting
+        libraries for use in Impala queries.
+      </p>
+
+      <p>
+        Impala ships with a sample build environment for UDFs, that you can study, experiment with, and adapt for
+        your own use. This sample build environment starts with the <cmdname>cmake</cmdname> configuration command,
+        which reads the file <filepath>CMakeLists.txt</filepath> and generates a <filepath>Makefile</filepath>
+        customized for your particular directory paths. Then the <cmdname>make</cmdname> command runs the actual
+        build steps based on the rules in the <filepath>Makefile</filepath>.
+      </p>
+
+      <p>
+        Impala loads the shared library from an HDFS location. After building a shared library containing one or
+        more UDFs, use <codeph>hdfs dfs</codeph> or <codeph>hadoop fs</codeph> commands to copy the binary file to
+        an HDFS location readable by Impala.
+      </p>
+
+      <p>
+        The final step in deployment is to issue a <codeph>CREATE FUNCTION</codeph> statement in the
+        <cmdname>impala-shell</cmdname> interpreter to make Impala aware of the new function. See
+        <xref href="impala_create_function.xml#create_function"/> for syntax details. Because each function is
+        associated with a particular database, always issue a <codeph>USE</codeph> statement to the appropriate
+        database before creating a function, or specify a fully qualified name, that is, <codeph>CREATE FUNCTION
+        <varname>db_name</varname>.<varname>function_name</varname></codeph>.
+      </p>
+
+      <p>
+        As you update the UDF code and redeploy updated versions of a shared library, use <codeph>DROP
+        FUNCTION</codeph> and <codeph>CREATE FUNCTION</codeph> to let Impala pick up the latest version of the
+        code.
+      </p>
+
+      <note>
+        <p conref="../shared/impala_common.xml#common/udf_persistence_restriction"/>
+        <p>
+          See <xref href="impala_create_function.xml#create_function"/> and <xref href="impala_drop_function.xml#drop_function"/>
+          for the new syntax for the persistent Java UDFs.
+        </p>
+      </note>
+
+      <p>
+        Prerequisites for the build environment are:
+      </p>
+
+<codeblock rev="CDH-37080"># Use the appropriate package installation command for your Linux distribution.
+sudo yum install gcc-c++ cmake boost-devel
+sudo yum install impala-udf-devel
+# The package name on Ubuntu and Debian is impala-udf-dev.
+</codeblock>
+
+      <p>
+        Then, unpack the sample code in <filepath>udf_samples.tar.gz</filepath> and use that as a template to set
+        up your build environment.
+      </p>
+
+      <p>
+        To build the original samples:
+      </p>
+
+<codeblock># Process CMakeLists.txt and set up appropriate Makefiles.
+cmake .
+# Generate shared libraries from UDF and UDAF sample code,
+# udf_samples/libudfsample.so and udf_samples/libudasample.so
+make</codeblock>
+
+      <p>
+        The sample code to examine, experiment with, and adapt is in these files:
+      </p>
+
+      <ul>
+        <li>
+          <filepath>udf-sample.h</filepath>: Header file that declares the signature for a scalar UDF
+          (<codeph>AddUDF</codeph>).
+        </li>
+
+        <li>
+          <filepath>udf-sample.cc</filepath>: Sample source for a simple UDF that adds two integers. Because
+          Impala can reference multiple function entry points from the same shared library, you could add other UDF
+          functions in this file and add their signatures to the corresponding header file.
+        </li>
+
+        <li>
+          <filepath>udf-sample-test.cc</filepath>: Basic unit tests for the sample UDF.
+        </li>
+
+        <li>
+          <filepath>uda-sample.h</filepath>: Header file that declares the signature for sample aggregate
+          functions. The SQL functions will be called <codeph>COUNT</codeph>, <codeph>AVG</codeph>, and
+          <codeph>STRINGCONCAT</codeph>. Because aggregate functions require more elaborate coding to handle the
+          processing for multiple phases, there are several underlying C++ functions such as
+          <codeph>CountInit</codeph>, <codeph>AvgUpdate</codeph>, and <codeph>StringConcatFinalize</codeph>.
+        </li>
+
+        <li>
+          <filepath>uda-sample.cc</filepath>: Sample source for simple UDAFs that demonstrate how to manage the
+          state transitions as the underlying functions are called during the different phases of query processing.
+          <ul>
+            <li>
+              The UDAF that imitates the <codeph>COUNT</codeph> function keeps track of a single incrementing
+              number; the merge functions combine the intermediate count values from each Impala node, and the
+              combined number is returned verbatim by the finalize function.
+            </li>
+
+            <li>
+              The UDAF that imitates the <codeph>AVG</codeph> function keeps track of two numbers, a count of rows
+              processed and the sum of values for a column. These numbers are updated and merged as with
+              <codeph>COUNT</codeph>, then the finalize function divides them to produce and return the final
+              average value.
+            </li>
+
+            <li>
+              The UDAF that concatenates string values into a comma-separated list demonstrates how to manage
+              storage for a string that increases in length as the function is called for multiple rows.
+            </li>
+          </ul>
+        </li>
+
+        <li>
+          <filepath>uda-sample-test.cc</filepath>: basic unit tests for the sample UDAFs.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="udf_performance">
+
+    <title>Performance Considerations for UDFs</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Performance"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        Because a UDF typically processes each row of a table, potentially being called billions of times, the
+        performance of each UDF is a critical factor in the speed of the overall ETL or ELT pipeline. Tiny
+        optimizations you can make within the function body can pay off in a big way when the function is called
+        over and over when processing a huge result set.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="udf_tutorial">
+
+    <title>Examples of Creating and Using UDFs</title>
+
+    <conbody>
+
+      <p>
+        This section demonstrates how to create and use all kinds of user-defined functions (UDFs).
+      </p>
+
+      <p>
+        For downloadable examples that you can experiment with, adapt, and use as templates for your own functions,
+        see <xref href="https://github.com/cloudera/impala-udf-samples" scope="external" format="html">the Cloudera
+        sample UDF github</xref>. You must have already installed the appropriate header files, as explained in
+        <xref href="impala_udf.xml#udf_demo_env"/>.
+      </p>
+
+<!-- Limitation: mini-TOC currently doesn't include the <example> tags. -->
+
+<!-- <p outputclass="toc inpage"/> -->
+
+      <example id="udf_sample_udf">
+
+        <title>Sample C++ UDFs: HasVowels, CountVowels, StripVowels</title>
+
+        <p>
+          This example shows 3 separate UDFs that operate on strings and return different data types. In the C++
+          code, the functions are <codeph>HasVowels()</codeph> (checks if a string contains any vowels),
+          <codeph>CountVowels()</codeph> (returns the number of vowels in a string), and
+          <codeph>StripVowels()</codeph> (returns a new string with vowels removed).
+        </p>
+
+        <p>
+          First, we add the signatures for these functions to <filepath>udf-sample.h</filepath> in the demo build
+          environment:
+        </p>
+
+<codeblock>BooleanVal HasVowels(FunctionContext* context, const StringVal&amp; input);
+IntVal CountVowels(FunctionContext* context, const StringVal&amp; arg1);
+StringVal StripVowels(FunctionContext* context, const StringVal&amp; arg1);</codeblock>
+
+        <p>
+          Then, we add the bodies of these functions to <filepath>udf-sample.cc</filepath>:
+        </p>
+
+<codeblock>BooleanVal HasVowels(FunctionContext* context, const StringVal&amp; input)
+{
+        if (input.is_null) return BooleanVal::null();
+
+        int index;
+        uint8_t *ptr;
+
+        for (ptr = input.ptr, index = 0; index &lt;= input.len; index++, ptr++)
+        {
+                uint8_t c = tolower(*ptr);
+                if (c == 'a' || c == 'e' || c == 'i' || c == 'o' || c == 'u')
+                {
+                        return BooleanVal(true);
+                }
+        }
+        return BooleanVal(false);
+}
+
+IntVal CountVowels(FunctionContext* context, const StringVal&amp; arg1)
+{
+        if (arg1.is_null) return IntVal::null();
+
+        int count;
+        int index;
+        uint8_t *ptr;
+
+        for (ptr = arg1.ptr, count = 0, index = 0; index &lt;= arg1.len; index++, ptr++)
+        {
+                uint8_t c = tolower(*ptr);
+                if (c == 'a' || c == 'e' || c == 'i' || c == 'o' || c == 'u')
+                {
+                        count++;
+                }
+        }
+        return IntVal(count);
+}
+
+StringVal StripVowels(FunctionContext* context, const StringVal&amp; arg1)
+{
+        if (arg1.is_null) return StringVal::null();
+
+        int index;
+        std::string original((const char *)arg1.ptr,arg1.len);
+        std::string shorter("");
+
+        for (index = 0; index &lt; original.length(); index++)
+        {
+                uint8_t c = original[index];
+                uint8_t l = tolower(c);
+
+                if (l == 'a' || l == 'e' || l == 'i' || l == 'o' || l == 'u')
+                {
+                        ;
+                }
+                else
+                {
+                    shorter.append(1, (char)c);
+                }
+        }
+// The modified string is stored in 'shorter', which is destroyed when this function ends. We need to make a string val
+// and copy the contents.
+        StringVal result(context, shorter.size()); // Only the version of the ctor that takes a context object allocates new memory
+        memcpy(result.ptr, shorter.c_str(), shorter.size());
+        return result;
+}</codeblock>
+
+        <p>
+          We build a shared library, <filepath>libudfsample.so</filepath>, and put the library file into HDFS
+          where Impala can read it:
+        </p>
+
+<codeblock>$ make
+[  0%] Generating udf_samples/uda-sample.ll
+[ 16%] Built target uda-sample-ir
+[ 33%] Built target udasample
+[ 50%] Built target uda-sample-test
+[ 50%] Generating udf_samples/udf-sample.ll
+[ 66%] Built target udf-sample-ir
+Scanning dependencies of target udfsample
+[ 83%] Building CXX object CMakeFiles/udfsample.dir/udf-sample.o
+Linking CXX shared library udf_samples/libudfsample.so
+[ 83%] Built target udfsample
+Linking CXX executable udf_samples/udf-sample-test
+[100%] Built target udf-sample-test
+$ hdfs dfs -put ./udf_samples/libudfsample.so /user/hive/udfs/libudfsample.so</codeblock>
+
+        <p>
+          Finally, we go into the <cmdname>impala-shell</cmdname> interpreter where we set up some sample data,
+          issue <codeph>CREATE FUNCTION</codeph> statements to set up the SQL function names, and call the
+          functions in some queries:
+        </p>
+
+<codeblock>[localhost:21000] &gt; create database udf_testing;
+[localhost:21000] &gt; use udf_testing;
+
+[localhost:21000] &gt; create function has_vowels (string) returns boolean location '/user/hive/udfs/libudfsample.so' symbol='HasVowels';
+[localhost:21000] &gt; select has_vowels('abc');
++------------------------+
+| udfs.has_vowels('abc') |
++------------------------+
+| true                   |
++------------------------+
+Returned 1 row(s) in 0.13s
+[localhost:21000] &gt; select has_vowels('zxcvbnm');
++----------------------------+
+| udfs.has_vowels('zxcvbnm') |
++----------------------------+
+| false                      |
++----------------------------+
+Returned 1 row(s) in 0.12s
+[localhost:21000] &gt; select has_vowels(null);
++-----------------------+
+| udfs.has_vowels(null) |
++-----------------------+
+| NULL                  |
++-----------------------+
+Returned 1 row(s) in 0.11s
+[localhost:21000] &gt; select s, has_vowels(s) from t2;
++-----------+--------------------+
+| s         | udfs.has_vowels(s) |
++-----------+--------------------+
+| lower     | true               |
+| UPPER     | true               |
+| Init cap  | true               |
+| CamelCase | true               |
++-----------+--------------------+
+Returned 4 row(s) in 0.24s
+
+[localhost:21000] &gt; create function count_vowels (string) returns int location '/user/hive/udfs/libudfsample.so' symbol='CountVowels';
+[localhost:21000] &gt; select count_vowels('cat in the hat');
++-------------------------------------+
+| udfs.count_vowels('cat in the hat') |
++-------------------------------------+
+| 4                                   |
++-------------------------------------+
+Returned 1 row(s) in 0.12s
+[localhost:21000] &gt; select s, count_vowels(s) from t2;
++-----------+----------------------+
+| s         | udfs.count_vowels(s) |
++-----------+----------------------+
+| lower     | 2                    |
+| UPPER     | 2                    |
+| Init cap  | 3                    |
+| CamelCase | 4                    |
++-----------+----------------------+
+Returned 4 row(s) in 0.23s
+[localhost:21000] &gt; select count_vowels(null);
++-------------------------+
+| udfs.count_vowels(null) |
++-------------------------+
+| NULL                    |
++-------------------------+
+Returned 1 row(s) in 0.12s
+
+[localhost:21000] &gt; create function strip_vowels (string) returns string location '/user/hive/udfs/libudfsample.so' symbol='StripVowels';
+[localhost:21000] &gt; select strip_vowels('abcdefg');
++------------------------------+
+| udfs.strip_vowels('abcdefg') |
++------------------------------+
+| bcdfg                        |
++------------------------------+
+Returned 1 row(s) in 0.11s
+[localhost:21000] &gt; select strip_vowels('ABCDEFG');
++------------------------------+
+| udfs.strip_vowels('abcdefg') |
++------------------------------+
+| BCDFG                        |
++------------------------------+
+Returned 1 row(s) in 0.12s
+[localhost:21000] &gt; select strip_vowels(null);
++-------------------------+
+| udfs.strip_vowels(null) |
++-------------------------+
+| NULL                    |
++-------------------------+
+Returned 1 row(s) in 0.16s
+[localhost:21000] &gt; select s, strip_vowels(s) from t2;
++-----------+----------------------+
+| s         | udfs.strip_vowels(s) |
++-----------+----------------------+
+| lower     | lwr                  |
+| UPPER     | PPR                  |
+| Init cap  | nt cp                |
+| CamelCase | CmlCs                |
++-----------+----------------------+
+Returned 4 row(s) in 0.24s</codeblock>
+
+      </example>
+
+      <example id="udf_sample_uda">
+
+        <title>Sample C++ UDA: SumOfSquares</title>
+
+        <p>
+          This example demonstrates a user-defined aggregate function (UDA) that produces the sum of the squares of
+          its input values.
+        </p>
+
+        <p>
+          The coding for a UDA is a little more involved than a scalar UDF, because the processing is split into
+          several phases, each implemented by a different function. Each phase is relatively straightforward: the
+          <q>update</q> and <q>merge</q> phases, where most of the work is done, read an input value and combine it
+          with some accumulated intermediate value.
+        </p>
+
+        <p>
+          As in our sample UDF from the previous example, we add function signatures to a header file (in this
+          case, <filepath>uda-sample.h</filepath>). Because this is a math-oriented UDA, we make two versions of
+          each function, one accepting an integer value and the other accepting a floating-point value.
+        </p>
+
+<codeblock>void SumOfSquaresInit(FunctionContext* context, BigIntVal* val);
+void SumOfSquaresInit(FunctionContext* context, DoubleVal* val);
+
+void SumOfSquaresUpdate(FunctionContext* context, const BigIntVal&amp; input, BigIntVal* val);
+void SumOfSquaresUpdate(FunctionContext* context, const DoubleVal&amp; input, DoubleVal* val);
+
+void SumOfSquaresMerge(FunctionContext* context, const BigIntVal&amp; src, BigIntVal* dst);
+void SumOfSquaresMerge(FunctionContext* context, const DoubleVal&amp; src, DoubleVal* dst);
+
+BigIntVal SumOfSquaresFinalize(FunctionContext* context, const BigIntVal&amp; val);
+DoubleVal SumOfSquaresFinalize(FunctionContext* context, const DoubleVal&amp; val);</codeblock>
+
+        <p>
+          We add the function bodies to a C++ source file (in this case, <filepath>uda-sample.cc</filepath>):
+        </p>
+
+<codeblock>void SumOfSquaresInit(FunctionContext* context, BigIntVal* val) {
+  val-&gt;is_null = false;
+  val-&gt;val = 0;
+}
+void SumOfSquaresInit(FunctionContext* context, DoubleVal* val) {
+  val-&gt;is_null = false;
+  val-&gt;val = 0.0;
+}
+
+void SumOfSquaresUpdate(FunctionContext* context, const BigIntVal&amp; input, BigIntVal* val) {
+  if (input.is_null) return;
+  val-&gt;val += input.val * input.val;
+}
+void SumOfSquaresUpdate(FunctionContext* context, const DoubleVal&amp; input, DoubleVal* val) {
+  if (input.is_null) return;
+  val-&gt;val += input.val * input.val;
+}
+
+void SumOfSquaresMerge(FunctionContext* context, const BigIntVal&amp; src, BigIntVal* dst) {
+  dst-&gt;val += src.val;
+}
+void SumOfSquaresMerge(FunctionContext* context, const DoubleVal&amp; src, DoubleVal* dst) {
+  dst-&gt;val += src.val;
+}
+
+BigIntVal SumOfSquaresFinalize(FunctionContext* context, const BigIntVal&amp; val) {
+  return val;
+}
+DoubleVal SumOfSquaresFinalize(FunctionContext* context, const DoubleVal&amp; val) {
+  return val;
+}</codeblock>
+
+        <p>
+          As with the sample UDF, we build a shared library and put it into HDFS:
+        </p>
+
+<codeblock>$ make
+[  0%] Generating udf_samples/uda-sample.ll
+[ 16%] Built target uda-sample-ir
+Scanning dependencies of target udasample
+[ 33%] Building CXX object CMakeFiles/udasample.dir/uda-sample.o
+Linking CXX shared library udf_samples/libudasample.so
+[ 33%] Built target udasample
+Scanning dependencies of target uda-sample-test
+[ 50%] Building CXX object CMakeFiles/uda-sample-test.dir/uda-sample-test.o
+Linking CXX executable udf_samples/uda-sample-test
+[ 50%] Built target uda-sample-test
+[ 50%] Generating udf_samples/udf-sample.ll
+[ 66%] Built target udf-sample-ir
+[ 83%] Built target udfsample
+[100%] Built target udf-sample-test
+$ hdfs dfs -put ./udf_samples/libudasample.so /user/hive/udfs/libudasample.so</codeblock>
+
+        <p>
+          To create the SQL function, we issue a <codeph>CREATE AGGREGATE FUNCTION</codeph> statement and specify
+          the underlying C++ function names for the different phases:
+        </p>
+
+<codeblock>[localhost:21000] &gt; use udf_testing;
+
+[localhost:21000] &gt; create table sos (x bigint, y double);
+[localhost:21000] &gt; insert into sos values (1, 1.1), (2, 2.2), (3, 3.3), (4, 4.4);
+Inserted 4 rows in 1.10s
+
+[localhost:21000] &gt; create aggregate function sum_of_squares(bigint) returns bigint
+  &gt; location '/user/hive/udfs/libudasample.so'
+  &gt; init_fn='SumOfSquaresInit'
+  &gt; update_fn='SumOfSquaresUpdate'
+  &gt; merge_fn='SumOfSquaresMerge'
+  &gt; finalize_fn='SumOfSquaresFinalize';
+
+[localhost:21000] &gt; -- Compute the same value using literals or the UDA;
+[localhost:21000] &gt; select 1*1 + 2*2 + 3*3 + 4*4;
++-------------------------------+
+| 1 * 1 + 2 * 2 + 3 * 3 + 4 * 4 |
++-------------------------------+
+| 30                            |
++-------------------------------+
+Returned 1 row(s) in 0.12s
+[localhost:21000] &gt; select sum_of_squares(x) from sos;
++------------------------+
+| udfs.sum_of_squares(x) |
++------------------------+
+| 30                     |
++------------------------+
+Returned 1 row(s) in 0.35s</codeblock>
+
+        <p>
+          Until we create the overloaded version of the UDA, it can only handle a single data type. To allow it to
+          handle <codeph>DOUBLE</codeph> as well as <codeph>BIGINT</codeph>, we issue another <codeph>CREATE
+          AGGREGATE FUNCTION</codeph> statement:
+        </p>
+
+<codeblock>[localhost:21000] &gt; select sum_of_squares(y) from sos;
+ERROR: AnalysisException: No matching function with signature: udfs.sum_of_squares(DOUBLE).
+
+[localhost:21000] &gt; create aggregate function sum_of_squares(double) returns double
+  &gt; location '/user/hive/udfs/libudasample.so'
+  &gt; init_fn='SumOfSquaresInit'
+  &gt; update_fn='SumOfSquaresUpdate'
+  &gt; merge_fn='SumOfSquaresMerge'
+  &gt; finalize_fn='SumOfSquaresFinalize';
+
+[localhost:21000] &gt; -- Compute the same value using literals or the UDA;
+[localhost:21000] &gt; select 1.1*1.1 + 2.2*2.2 + 3.3*3.3 + 4.4*4.4;
++-----------------------------------------------+
+| 1.1 * 1.1 + 2.2 * 2.2 + 3.3 * 3.3 + 4.4 * 4.4 |
++-----------------------------------------------+
+| 36.3                                          |
++-----------------------------------------------+
+Returned 1 row(s) in 0.12s
+[localhost:21000] &gt; select sum_of_squares(y) from sos;
++------------------------+
+| udfs.sum_of_squares(y) |
++------------------------+
+| 36.3                   |
++------------------------+
+Returned 1 row(s) in 0.35s</codeblock>
+
+        <p>
+          Typically, you use a UDA in queries with <codeph>GROUP BY</codeph> clauses, to produce a result set with
+          a separate aggregate value for each combination of values from the <codeph>GROUP BY</codeph> clause.
+          Let's change our sample table to use <codeph>0</codeph> to indicate rows containing even values, and
+          <codeph>1</codeph> to flag rows containing odd values. Then the <codeph>GROUP BY</codeph> query can
+          return two values, the sum of the squares for the even values, and the sum of the squares for the odd
+          values:
+        </p>
+
+<codeblock>[localhost:21000] &gt; insert overwrite sos values (1, 1), (2, 0), (3, 1), (4, 0);
+Inserted 4 rows in 1.24s
+
+[localhost:21000] &gt; -- Compute 1 squared + 3 squared, and 2 squared + 4 squared;
+[localhost:21000] &gt; select y, sum_of_squares(x) from sos group by y;
++---+------------------------+
+| y | udfs.sum_of_squares(x) |
++---+------------------------+
+| 1 | 10                     |
+| 0 | 20                     |
++---+------------------------+
+Returned 2 row(s) in 0.43s</codeblock>
+
+      </example>
+    </conbody>
+  </concept>
+
+  <concept id="udf_security">
+
+    <title>Security Considerations for User-Defined Functions</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Security"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        When the Impala authorization feature is enabled:
+      </p>
+
+      <ul>
+        <li>
+          To call a UDF in a query, you must have the required read privilege for any databases and tables used in
+          the query.
+        </li>
+
+        <li>
+          Because incorrectly coded UDFs could cause performance or capacity problems, for example by going into
+          infinite loops or allocating excessive amounts of memory, only an administrative user can create UDFs.
+          That is, to execute the <codeph>CREATE FUNCTION</codeph> statement requires the <codeph>ALL</codeph>
+          privilege on the server.
+        </li>
+      </ul>
+
+      <p>
+        See <xref href="impala_authorization.xml#authorization"/> for details about authorization in Impala.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="udf_limits">
+
+    <title>Limitations and Restrictions for Impala UDFs</title>
+
+    <conbody>
+
+      <p>
+        The following limitations and restrictions apply to Impala UDFs in the current release:
+      </p>
+
+      <ul>
+        <li>
+          Impala does not support Hive UDFs that accept or return composite or nested types, or other types not
+          available in Impala tables.
+        </li>
+
+        <li>
+          <p conref="../shared/impala_common.xml#common/current_user_caveat"/>
+        </li>
+
+        <li>
+          All Impala UDFs must be deterministic, that is, produce the same output each time when passed the same
+          argument values. For example, an Impala UDF must not call functions such as <codeph>rand()</codeph> to
+          produce different values for each invocation. It must not retrieve data from external sources, such as
+          from disk or over the network.
+        </li>
+
+        <li>
+          An Impala UDF must not spawn other threads or processes.
+        </li>
+
+        <li rev="2.5.0 IMPALA-2843">
+          Prior to <keyword keyref="impala25_full"/> when the <cmdname>catalogd</cmdname> process is restarted,
+          all UDFs become undefined and must be reloaded. In <keyword keyref="impala25_full"/> and higher, this
+          limitation only applies to older Java UDFs. Re-create those UDFs using the new
+          <codeph>CREATE FUNCTION</codeph> syntax for Java UDFs, which excludes the function signature,
+          to remove the limitation entirely.
+        </li>
+
+        <li>
+          Impala currently does not support user-defined table functions (UDTFs).
+        </li>
+
+        <li rev="2.0.0">
+          The <codeph>CHAR</codeph> and <codeph>VARCHAR</codeph> types cannot be used as input arguments or return
+          values for UDFs.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+</concept>


[48/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/shared/impala_common.xml
----------------------------------------------------------------------
diff --git a/docs/shared/impala_common.xml b/docs/shared/impala_common.xml
new file mode 100644
index 0000000..b0c9ee5
--- /dev/null
+++ b/docs/shared/impala_common.xml
@@ -0,0 +1,3690 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="common">
+
+  <title>Reusable Text, Paragraphs, List Items, and Other Elements for Impala</title>
+
+  <conbody>
+
+    <p>
+      All the elements in this file with IDs are intended to be conref'ed elsewhere. Practically all of the
+      conref'ed elements for the Impala docs are in this file, to avoid questions of when it's safe to remove or
+      move something in any of the 'main' files, and avoid having to change and conref references as a result.
+    </p>
+
+    <p>
+      This file defines some dummy subheadings as section elements, just for self-documentation. Using sections
+      instead of nested concepts lets all the conref links point to a very simple name pattern,
+      '#common/id_within_the_file', rather than a 3-part reference with an intervening, variable concept ID.
+    </p>
+
+    <section id="concepts">
+
+      <title>Conceptual Content</title>
+
+      <p>
+        Overview and conceptual information for Impala as a whole.
+      </p>
+
+      <!-- Reconcile the 'advantages' and 'benefits' elements; be mindful of where each is used. -->
+
+      <p id="impala_advantages">
+        The following are some of the key advantages of Impala:
+
+        <ul>
+          <li>
+            Impala integrates with the existing <keyword keyref="hadoop_distro"/> ecosystem, meaning data can be stored, shared, and accessed using
+            the various solutions included with <keyword keyref="hadoop_distro"/>. This also avoids data silos and minimizes expensive data movement.
+          </li>
+
+          <li>
+            Impala provides access to data stored in <keyword keyref="hadoop_distro"/> without requiring the Java skills required for MapReduce jobs.
+            Impala can access data directly from the HDFS file system. Impala also provides a SQL front-end to access
+            data in the HBase database system, <ph rev="2.2.0">or in the Amazon Simple Storage System (S3)</ph>.
+          </li>
+
+          <li>
+            Impala returns results typically within seconds or a few minutes, rather than the many minutes or hours
+            that are often required for Hive queries to complete.
+          </li>
+
+          <li>
+            Impala is pioneering the use of the Parquet file format, a columnar storage layout that is optimized for
+            large-scale queries typical in data warehouse scenarios.
+          </li>
+        </ul>
+      </p>
+
+      <p id="impala_benefits">
+        Impala provides:
+
+        <ul>
+          <li>
+            Familiar SQL interface that data scientists and analysts already know.
+          </li>
+
+          <li>
+            Ability to query high volumes of data (<q>big data</q>) in Apache Hadoop.
+          </li>
+
+          <li>
+            Distributed queries in a cluster environment, for convenient scaling and to make use of cost-effective
+            commodity hardware.
+          </li>
+
+          <li>
+            Ability to share data files between different components with no copy or export/import step; for example,
+            to write with Pig, transform with Hive and query with Impala. Impala can read from and write to Hive
+            tables, enabling simple data interchange using Impala for analytics on Hive-produced data.
+          </li>
+
+          <li>
+            Single system for big data processing and analytics, so customers can avoid costly modeling and ETL just
+            for analytics.
+          </li>
+        </ul>
+      </p>
+
+    </section>
+
+    <section id="sentry">
+
+      <title>Sentry-Related Content</title>
+
+      <p>
+        Material related to Sentry security, intended to be reused between Hive and Impala. Complicated by the fact
+        that most of it will probably be multi-paragraph or involve subheads, might need to be represented as
+        nested topics at the end of this file.
+      </p>
+
+      <table id="sentry_privileges_objects">
+        <title>Valid privilege types and objects they apply to</title>
+        <tgroup cols="2">
+          <colspec colnum="1" colname="col1" colwidth="1*"/>
+          <colspec colnum="2" colname="col2" colwidth="2*"/>
+          <thead>
+            <row>
+              <entry><b>Privilege</b></entry>
+              <entry><b>Object</b></entry>
+            </row>
+          </thead>
+          <tbody>
+            <row>
+              <entry>INSERT</entry>
+              <entry>DB, TABLE</entry>
+            </row>
+            <row>
+              <entry>SELECT</entry>
+              <entry>DB, TABLE, COLUMN</entry>
+            </row>
+            <row>
+              <entry>ALL</entry>
+              <entry>SERVER, TABLE, DB, URI</entry>
+            </row>
+          </tbody>
+        </tgroup>
+      </table>
+
+      <table id="privileges_sql">
+        <title>Privilege table for Hive &amp; Impala operations</title>
+        <tgroup cols="4">
+          <colspec colnum="1" colname="col1" colwidth="1.31*"/>
+          <colspec colnum="2" colname="col2" colwidth="1.17*"/>
+          <colspec colnum="3" colname="col3" colwidth="1*"/>
+          <colspec colname="newCol4" colnum="4" colwidth="1*"/>
+          <thead>
+            <row>
+              <entry>Operation</entry>
+              <entry>Scope</entry>
+              <entry>Privileges Required</entry>
+              <entry>URI</entry>
+            </row>
+          </thead>
+          <tbody>
+            <row id="create_database_privs">
+              <entry>CREATE DATABASE</entry>
+              <entry>SERVER</entry>
+              <entry>ALL</entry>
+              <entry/>
+            </row>
+            <row id="drop_database_privs">
+              <entry>DROP DATABASE</entry>
+              <entry>DATABASE</entry>
+              <entry>ALL</entry>
+              <entry/>
+            </row>
+            <row id="create_table_privs">
+              <entry>CREATE TABLE</entry>
+              <entry>DATABASE</entry>
+              <entry>ALL</entry>
+              <entry/>
+            </row>
+            <row id="drop_table_privs">
+              <entry>DROP TABLE</entry>
+              <entry>TABLE</entry>
+              <entry>ALL</entry>
+              <entry/>
+            </row>
+            <row id="create_view_privs">
+              <entry>CREATE VIEW<p>-This operation is allowed if you have
+                  column-level <codeph>SELECT</codeph> access to the columns
+                  being used.</p></entry>
+              <entry>DATABASE; SELECT on TABLE; </entry>
+              <entry>ALL</entry>
+              <entry/>
+            </row>
+            <row>
+              <entry>ALTER VIEW<p>-This operation is allowed if you have
+                  column-level <codeph>SELECT</codeph> access to the columns
+                  being used.</p></entry>
+              <entry>VIEW/TABLE</entry>
+              <entry>ALL</entry>
+              <entry/>
+            </row>
+            <row id="drop_view_privs">
+              <entry>DROP VIEW</entry>
+              <entry>VIEW/TABLE</entry>
+              <entry>ALL</entry>
+              <entry/>
+            </row>
+            <row id="alter_table_add_columns_privs">
+              <entry>ALTER TABLE .. ADD COLUMNS</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry/>
+            </row>
+            <row id="alter_table_replace_columns_privs">
+              <entry>ALTER TABLE .. REPLACE COLUMNS</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry/>
+            </row>
+            <row id="alter_table_change_column_privs">
+              <entry>ALTER TABLE .. CHANGE column</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry/>
+            </row>
+            <row id="alter_table_rename_privs">
+              <entry>ALTER TABLE .. RENAME</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry/>
+            </row>
+            <row id="alter_table_set_tblproperties_privs">
+              <entry>ALTER TABLE .. SET TBLPROPERTIES</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry/>
+            </row>
+            <row id="alter_table_set_fileformat_privs">
+              <entry>ALTER TABLE .. SET FILEFORMAT</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry/>
+            </row>
+            <row id="alter_table_set_location_privs">
+              <entry>ALTER TABLE .. SET LOCATION</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry>URI</entry>
+            </row>
+            <row id="alter_table_add_partition_privs">
+              <entry>ALTER TABLE .. ADD PARTITION</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry/>
+            </row>
+            <row id="alter_table_add_partition_location_privs">
+              <entry>ALTER TABLE .. ADD PARTITION location</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry>URI</entry>
+            </row>
+            <row id="alter_table_drop_partition_privs">
+              <entry>ALTER TABLE .. DROP PARTITION</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry/>
+            </row>
+            <row id="alter_table_partition_set_fileformat_privs">
+              <entry>ALTER TABLE .. PARTITION SET FILEFORMAT</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry/>
+            </row>
+            <row id="show_create_table_privs">
+              <entry>SHOW CREATE TABLE</entry>
+              <entry>TABLE</entry>
+              <entry>SELECT/INSERT</entry>
+              <entry/>
+            </row>
+            <row id="show_partitions_privs">
+              <entry>SHOW PARTITIONS</entry>
+              <entry>TABLE</entry>
+              <entry>SELECT/INSERT</entry>
+              <entry/>
+            </row>
+            <row>
+              <entry>SHOW TABLES<p>-Output includes all the tables for which
+                  the user has table-level privileges and all the tables for
+                  which the user has some column-level privileges.</p></entry>
+              <entry>TABLE</entry>
+              <entry>SELECT/INSERT</entry>
+              <entry/>
+            </row>
+            <row>
+              <entry>SHOW GRANT ROLE<p>-Output includes an additional field
+                  for any column-level privileges.</p></entry>
+              <entry>TABLE</entry>
+              <entry>SELECT/INSERT</entry>
+              <entry/>
+            </row>
+            <row id="describe_table_privs">
+              <entry>DESCRIBE TABLE<p>-Output shows <i>all</i> columns if the
+                  user has table level-privileges or <codeph>SELECT</codeph>
+                  privilege on at least one table column</p></entry>
+              <entry>TABLE</entry>
+              <entry>SELECT/INSERT</entry>
+              <entry/>
+            </row>
+            <row id="load_data_privs">
+              <entry>LOAD DATA</entry>
+              <entry>TABLE</entry>
+              <entry>INSERT</entry>
+              <entry>URI</entry>
+            </row>
+            <row id="select_privs">
+              <entry>SELECT<p>-You can grant the SELECT privilege on a view to
+                  give users access to specific columns of a table they do not
+                  otherwise have access to.</p><p>-See
+                  <xref audience="integrated" href="../topics/sg_hive_sql.xml#concept_c2q_4qx_p4/col_level_auth_sentry"/><xref audience="standalone" href="https://www.cloudera.com/documentation/enterprise/latest/topics/sg_hive_sql.html" format="html" scope="external"/>
+                  for details on allowed column-level
+                operations.</p></entry>
+              <entry>VIEW/TABLE; COLUMN</entry>
+              <entry>SELECT</entry>
+              <entry/>
+            </row>
+            <row id="insert_overwrite_table_privs">
+              <entry>INSERT OVERWRITE TABLE</entry>
+              <entry>TABLE</entry>
+              <entry>INSERT</entry>
+              <entry/>
+            </row>
+            <row id="create_table_as_select_privs">
+              <entry>CREATE TABLE .. AS SELECT<p>-This operation is allowed if
+                  you have column-level <codeph>SELECT</codeph> access to the
+                  columns being used.</p></entry>
+              <entry>DATABASE; SELECT on TABLE</entry>
+              <entry>ALL</entry>
+              <entry/>
+            </row>
+            <row id="use_privs">
+              <entry>USE &lt;dbName&gt;</entry>
+              <entry>Any</entry>
+              <entry/>
+              <entry/>
+            </row>
+            <row id="create_function_privs">
+              <entry>CREATE FUNCTION</entry>
+              <entry>SERVER</entry>
+              <entry>ALL</entry>
+              <entry/>
+            </row>
+            <row id="alter_table_set_serdeproperties_privs">
+              <entry>ALTER TABLE .. SET SERDEPROPERTIES</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry/>
+            </row>
+            <row>
+              <entry>ALTER TABLE .. PARTITION SET SERDEPROPERTIES</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry/>
+            </row>
+            <row id="_privs">
+              <entry namest="col1" nameend="newCol4"><b>Hive-Only
+                  Operations</b></entry>
+            </row>
+            <row id="insert_overwrite_directory_privs">
+              <entry>INSERT OVERWRITE DIRECTORY</entry>
+              <entry>TABLE</entry>
+              <entry>INSERT</entry>
+              <entry>URI</entry>
+            </row>
+            <row id="analyze_table_privs">
+              <entry>Analyze TABLE</entry>
+              <entry>TABLE</entry>
+              <entry>SELECT + INSERT</entry>
+              <entry/>
+            </row>
+            <row id="import_table_privs">
+              <entry>IMPORT TABLE</entry>
+              <entry>DATABASE</entry>
+              <entry>ALL</entry>
+              <entry>URI</entry>
+            </row>
+            <row id="export_table_privs">
+              <entry>EXPORT TABLE</entry>
+              <entry>TABLE</entry>
+              <entry>SELECT</entry>
+              <entry>URI</entry>
+            </row>
+            <row id="alter_table_touch_privs">
+              <entry>ALTER TABLE TOUCH</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry/>
+            </row>
+            <row id="alter_table_touch_partition_privs">
+              <entry>ALTER TABLE TOUCH PARTITION</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry/>
+            </row>
+            <row id="alter_table_clustered_by_privs">
+              <entry>ALTER TABLE .. CLUSTERED BY SORTED BY</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry/>
+            </row>
+            <row id="alter_table_enable_privs">
+              <entry>ALTER TABLE .. ENABLE/DISABLE</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry/>
+            </row>
+            <row id="alter_table_partition_enable_privs">
+              <entry>ALTER TABLE .. PARTITION ENABLE/DISABLE</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry/>
+            </row>
+            <row id="alter_table_partition_rename_privs">
+              <entry>ALTER TABLE .. PARTITION.. RENAME TO PARTITION</entry>
+              <entry>TABLE</entry>
+              <entry>ALL on DATABASE</entry>
+              <entry/>
+            </row>
+            <row>
+              <entry>MSCK REPAIR TABLE</entry>
+              <entry>TABLE</entry>
+              <entry>ALL</entry>
+              <entry/>
+            </row>
+            <row id="alter_database_privs">
+              <entry>ALTER DATABASE</entry>
+              <entry>DATABASE</entry>
+              <entry>ALL</entry>
+              <entry/>
+            </row>
+            <row id="describe_database_privs">
+              <entry>DESCRIBE DATABASE</entry>
+              <entry>DATABASE</entry>
+              <entry>SELECT/INSERT</entry>
+              <entry/>
+            </row>
+            <row id="show_columns_privs">
+              <entry>SHOW COLUMNS<p>-Output for this operation filters columns
+                  to which the user does not have explicit
+                    <codeph>SELECT</codeph> access </p></entry>
+              <entry>TABLE</entry>
+              <entry>SELECT/INSERT</entry>
+              <entry/>
+            </row>
+            <row id="create_index_privs">
+              <entry>CREATE INDEX</entry>
+              <entry>TABLE</entry>
+              <entry>ALL</entry>
+              <entry/>
+            </row>
+            <row id="drop_index_privs">
+              <entry>DROP INDEX</entry>
+              <entry>TABLE</entry>
+              <entry>ALL</entry>
+              <entry/>
+            </row>
+            <row id="show_indexes_privs">
+              <entry>SHOW INDEXES</entry>
+              <entry>TABLE</entry>
+              <entry>SELECT/INSERT</entry>
+              <entry/>
+            </row>
+            <row id="grant_privs">
+              <entry>GRANT PRIVILEGE</entry>
+              <entry>Allowed only for Sentry admin users</entry>
+              <entry/>
+              <entry/>
+            </row>
+            <row id="revoke_privs">
+              <entry>REVOKE PRIVILEGE</entry>
+              <entry>Allowed only for Sentry admin users</entry>
+              <entry/>
+              <entry/>
+            </row>
+            <row id="show_grants_privs">
+              <entry>SHOW GRANTS</entry>
+              <entry>Allowed only for Sentry admin users</entry>
+              <entry/>
+              <entry/>
+            </row>
+            <row id="show_tblproperties_privs">
+              <entry>SHOW TBLPROPERTIES</entry>
+              <entry>TABLE</entry>
+              <entry>SELECT/INSERT</entry>
+              <entry/>
+            </row>
+            <row id="describe_table_partition_privs">
+              <entry>DESCRIBE TABLE .. PARTITION</entry>
+              <entry>TABLE</entry>
+              <entry>SELECT/INSERT</entry>
+              <entry/>
+            </row>
+            <row id="add_jar_privs">
+              <entry>ADD JAR</entry>
+              <entry>Not Allowed</entry>
+              <entry/>
+              <entry/>
+            </row>
+            <row id="add_file_privs">
+              <entry>ADD FILE</entry>
+              <entry>Not Allowed</entry>
+              <entry/>
+              <entry/>
+            </row>
+            <row id="dfs_privs">
+              <entry>DFS</entry>
+              <entry>Not Allowed</entry>
+              <entry/>
+              <entry/>
+            </row>
+            <row>
+              <entry namest="col1" nameend="newCol4"><b>Impala-Only
+                  Operations</b></entry>
+            </row>
+            <row id="explain_privs">
+              <entry>EXPLAIN</entry>
+              <entry>TABLE; COLUMN</entry>
+              <entry>SELECT</entry>
+              <entry/>
+            </row>
+            <row id="invalidate_metadata_privs">
+              <entry>INVALIDATE METADATA</entry>
+              <entry>SERVER</entry>
+              <entry>ALL</entry>
+              <entry/>
+            </row>
+            <row id="invalidate_metadata_table_privs">
+              <entry>INVALIDATE METADATA &lt;table name&gt;</entry>
+              <entry>TABLE</entry>
+              <entry>SELECT/INSERT</entry>
+              <entry/>
+            </row>
+            <row id="refresh_privs">
+              <entry>REFRESH &lt;table name&gt; or REFRESH &lt;table name&gt; PARTITION (&lt;partition_spec&gt;)</entry>
+              <entry>TABLE</entry>
+              <entry>SELECT/INSERT</entry>
+              <entry/>
+            </row>
+            <row id="drop_function_privs">
+              <entry>DROP FUNCTION</entry>
+              <entry>SERVER</entry>
+              <entry>ALL</entry>
+              <entry/>
+            </row>
+            <row id="compute_stats_privs">
+              <entry>COMPUTE STATS</entry>
+              <entry>TABLE</entry>
+              <entry>ALL</entry>
+              <entry/>
+            </row>
+          </tbody>
+        </tgroup>
+      </table>
+
+    <p rev="IMPALA-2660 CDH-40241" id="auth_to_local_instructions">
+      In <keyword keyref="impala26_full"/> and higher, Impala recognizes the <codeph>auth_to_local</codeph> setting,
+      specified through the HDFS configuration setting
+      <codeph>hadoop.security.auth_to_local</codeph>
+      or the Cloudera Manager setting
+      <uicontrol>Additional Rules to Map Kerberos Principals to Short Names</uicontrol>.
+      This feature is disabled by default, to avoid an unexpected change in security-related behavior.
+      To enable it:
+      <ul>
+        <li>
+          <p>
+            For clusters not managed by Cloudera Manager, specify <codeph>--load_auth_to_local_rules=true</codeph>
+            in the <cmdname>impalad</cmdname> and <cmdname>catalogd</cmdname>configuration settings.
+          </p>
+        </li>
+        <li>
+          <p>
+            For clusters managed by Cloudera Manager, select the 
+            <uicontrol>Use HDFS Rules to Map Kerberos Principals to Short Names</uicontrol>
+            checkbox to enable the service-wide <codeph>load_auth_to_local_rules</codeph> configuration setting.
+            Then restart the Impala service.
+          </p>
+        </li>
+      </ul>
+      See <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/sg_auth_to_local_isolate.html" scope="external" format="html">Using Auth-to-Local Rules to Isolate Cluster Users</xref> for general information about this feature.
+    </p>
+
+    <note id="authentication_vs_authorization">
+      Regardless of the authentication mechanism used, Impala always creates HDFS directories and data files
+      owned by the same user (typically <codeph>impala</codeph>). To implement user-level access to different
+      databases, tables, columns, partitions, and so on, use the Sentry authorization feature, as explained in
+      <xref href="../topics/impala_authorization.xml#authorization"/>.
+    </note>
+
+<!-- Contrived nesting needed to allow <ph> with ID to be reused inside the <title> of a conref. -->
+
+    <p>
+      <b><ph id="title_sentry_debug">Debugging Failed Sentry Authorization Requests</ph></b>
+    </p>
+
+    <p id="sentry_debug">
+      Sentry logs all facts that lead up to authorization decisions at the debug level. If you do not understand
+      why Sentry is denying access, the best way to debug is to temporarily turn on debug logging:
+      <ul>
+        <li>
+          In Cloudera Manager, add <codeph>log4j.logger.org.apache.sentry=DEBUG</codeph> to the logging settings
+          for your service through the corresponding <uicontrol>Logging Safety Valve</uicontrol> field for the
+          Impala, Hive Server 2, or Solr Server services.
+        </li>
+
+        <li>
+          On systems not managed by Cloudera Manager, add <codeph>log4j.logger.org.apache.sentry=DEBUG</codeph>
+          to the <filepath>log4j.properties</filepath> file on each host in the cluster, in the appropriate
+          configuration directory for each service.
+        </li>
+      </ul>
+      Specifically, look for exceptions and messages such as:
+<codeblock xml:space="preserve">FilePermission server..., RequestPermission server...., result [true|false]</codeblock>
+      which indicate each evaluation Sentry makes. The <codeph>FilePermission</codeph> is from the policy file,
+      while <codeph>RequestPermission</codeph> is the privilege required for the query. A
+      <codeph>RequestPermission</codeph> will iterate over all appropriate <codeph>FilePermission</codeph>
+      settings until a match is found. If no matching privilege is found, Sentry returns <codeph>false</codeph>
+      indicating <q>Access Denied</q> .
+<!--
+[1]
+Impala: Impala Daemon -> Advanced -> Impala Daemon Logging Safety Valve 
+Hive: Hive Server 2 -> Advanced -> HiveServer2 Logging Safety Valve 
+Search: Solr Server -> Advanced -> HiveServer2 Logging Safety Valve
+-->
+    </p>
+
+  </section>
+
+  <section id="cm">
+
+    <title>Cloudera Manager Terminology</title>
+
+    <p>
+      Especially during the transition from CM 4 to CM 5, we'll use some stock phraseology to talk about fields
+      and such. Also there are some task steps etc. to conref under the Impala Service page that are easier
+      to keep track of here instead of in cm_common_elements.xml. (Although as part of Apache work, anything
+      CM might naturally move out of this file.)
+    </p>
+
+    <p>
+      <ph id="safety_valve"> In Cloudera Manager 4, these fields are labelled <uicontrol>Safety
+      Valve</uicontrol>; in Cloudera Manager 5, they are called <uicontrol>Advanced Configuration
+      Snippet</uicontrol>. </ph>
+    </p>
+
+    <ul>
+      <li id="go_impala_service">Go to the Impala service.</li>
+      <li id="restart_impala_service">Restart the Impala service.</li>
+    </ul>
+
+  </section>
+
+  <section id="citi">
+
+    <title>Items from the Citibank Escalation Spreadsheet</title>
+
+    <p>
+      Paragraphs with IDs are intended to be reused both in the FAQ and the User's Guide. They refer to feature
+      requests or misunderstandings encountered by Citibank, captured in the escalation spreadsheet here:
+      <xref href="https://docs.google.com/a/cloudera.com/spreadsheet/ccc?key=0AplfwQJKyyTWdFdhY0E5WHVwNXZSTG9sMEZwQy1QZ1E&amp;usp=drive_web#gid=0" scope="external" format="html"/>.
+    </p>
+
+    <p id="string_concatenation">
+      With Impala, you use the built-in <codeph>CONCAT()</codeph> function to concatenate two, three, or more
+      strings:
+<codeblock xml:space="preserve">select concat('some prefix: ', col1) from t1;
+select concat('abc','mno','xyz');</codeblock>
+      Impala does not currently support operators for string concatenation, such as <codeph>||</codeph> as seen
+      in some other database systems.
+    </p>
+
+    <p id="column_aliases">
+      You can specify column aliases with or without the <codeph>AS</codeph> keyword, and with no quotation
+      marks, single quotation marks, or double quotation marks. Some kind of quotation marks are required if the
+      column alias contains any spaces or other problematic characters. The alias text is displayed in the
+      <cmdname>impala-shell</cmdname> output as all-lowercase. For example:
+<codeblock xml:space="preserve">[localhost:21000] &gt; select c1 First_Column from t;
+[localhost:21000] &gt; select c1 as First_Column from t;
++--------------+
+| first_column |
++--------------+
+...
+
+[localhost:21000] &gt; select c1 'First Column' from t;
+[localhost:21000] &gt; select c1 as 'First Column' from t;
++--------------+
+| first column |
++--------------+
+...
+
+[localhost:21000] &gt; select c1 "First Column" from t;
+[localhost:21000] &gt; select c1 as "First Column" from t;
++--------------+
+| first column |
++--------------+
+...</codeblock>
+    </p>
+
+    <p id="temp_tables">
+      Currently, Impala does not support temporary tables. Some other database systems have a class of
+      <q>lightweight</q> tables that are held only in memory and/or that are only accessible by one connection
+      and disappear when the session ends. In Impala, creating new databases is a relatively lightweight
+      operation, so as an alternative, you could create a database with a unique name and use <codeph>CREATE
+      TABLE LIKE</codeph>, <codeph>CREATE TABLE AS SELECT</codeph>, and <codeph>INSERT</codeph> statements to
+      create a table in that database to hold the result set of a query, to use in subsequent queries. When
+      finished, issue a <codeph>DROP TABLE</codeph> statement followed by <codeph>DROP DATABASE</codeph>.
+    </p>
+
+  </section>
+
+  <section id="standards">
+
+    <title>Blurbs About Standards Compliance</title>
+
+    <p>
+      The following blurbs simplify the process of flagging which SQL standard various features were first
+      introduced in. The wording and the tagging can be modified by editing one central instance of each blurb.
+      Not extensively used yet, just here and there in the SQL Language Reference section.
+    </p>
+
+    <p id="sql1986">
+<!-- No Wikipedia page for SQL-1986, so no link. -->
+      <b>Standards compliance:</b> Introduced in SQL-1986.
+    </p>
+
+    <p id="sql1989">
+<!-- No Wikipedia page for SQL-1989, so no link. -->
+      <b>Standards compliance:</b> Introduced in SQL-1989.
+    </p>
+
+    <p id="sql1992">
+      <b>Standards compliance:</b> Introduced in
+      <xref href="http://en.wikipedia.org/wiki/SQL-92" scope="external" format="html">SQL-1992</xref>.
+    </p>
+
+    <p id="sql1999">
+      <b>Standards compliance:</b> Introduced in
+      <xref href="http://en.wikipedia.org/wiki/SQL:1999" scope="external" format="html">SQL:1999</xref>.
+    </p>
+
+    <p id="sql2003">
+      <b>Standards compliance:</b> Introduced in
+      <xref href="http://en.wikipedia.org/wiki/SQL:2003" scope="external" format="html">SQL:2003</xref>.
+    </p>
+
+    <p id="sql2008">
+      <b>Standards compliance:</b> Introduced in
+      <xref href="http://en.wikipedia.org/wiki/SQL:2008" scope="external" format="html">SQL:2008</xref>.
+    </p>
+
+    <p id="sql2011">
+      <b>Standards compliance:</b> Introduced in
+      <xref href="http://en.wikipedia.org/wiki/SQL:2011" scope="external" format="html">SQL:2011</xref>.
+    </p>
+
+    <p id="hiveql">
+      <b>Standards compliance:</b> Extension first introduced in HiveQL.
+    </p>
+
+    <p id="impalaql">
+      <b>Standards compliance:</b> Extension first introduced in Impala.
+    </p>
+
+  </section>
+
+  <section id="refresh_invalidate">
+
+    <title>Background Info for REFRESH, INVALIDATE METADATA, and General Metadata Discussion</title>
+
+    <p id="invalidate_then_refresh" rev="DOCS-1013">
+      Because <codeph>REFRESH <varname>table_name</varname></codeph> only works for tables that the current
+      Impala node is already aware of, when you create a new table in the Hive shell, enter
+      <codeph>INVALIDATE METADATA <varname>new_table</varname></codeph> before you can see the new table in
+      <cmdname>impala-shell</cmdname>. Once the table is known by Impala, you can issue <codeph>REFRESH
+      <varname>table_name</varname></codeph> after you add data files for that table.
+    </p>
+
+    <p id="refresh_vs_invalidate">
+      <codeph>INVALIDATE METADATA</codeph> and <codeph>REFRESH</codeph> are counterparts: <codeph>INVALIDATE
+      METADATA</codeph> waits to reload the metadata when needed for a subsequent query, but reloads all the
+      metadata for the table, which can be an expensive operation, especially for large tables with many
+      partitions. <codeph>REFRESH</codeph> reloads the metadata immediately, but only loads the block location
+      data for newly added data files, making it a less expensive operation overall. If data was altered in some
+      more extensive way, such as being reorganized by the HDFS balancer, use <codeph>INVALIDATE
+      METADATA</codeph> to avoid a performance penalty from reduced local reads. If you used Impala version 1.0,
+      the <codeph>INVALIDATE METADATA</codeph> statement works just like the Impala 1.0 <codeph>REFRESH</codeph>
+      statement did, while the Impala 1.1 <codeph>REFRESH</codeph> is optimized for the common use case of adding
+      new data files to an existing table, thus the table name argument is now required.
+    </p>
+
+  </section>
+
+    <section id="sql_ref">
+
+      <title>SQL Language Reference Snippets</title>
+
+      <p>
+        These reusable chunks were taken from conrefs originally in <filepath>ciiu_langref_sql.xml</filepath>. Or
+        they are primarily used in new SQL syntax topics underneath that parent topic.
+      </p>
+
+<codeblock id="parquet_fallback_schema_resolution_example"><![CDATA[
+create database schema_evolution;
+use schema_evolution;
+create table t1 (c1 int, c2 boolean, c3 string, c4 timestamp)
+  stored as parquet;
+insert into t1 values
+  (1, true, 'yes', now()),
+  (2, false, 'no', now() + interval 1 day);
+
+select * from t1;
++----+-------+-----+-------------------------------+
+| c1 | c2    | c3  | c4                            |
++----+-------+-----+-------------------------------+
+| 1  | true  | yes | 2016-06-28 14:53:26.554369000 |
+| 2  | false | no  | 2016-06-29 14:53:26.554369000 |
++----+-------+-----+-------------------------------+
+
+desc formatted t1;
+...
+| Location:   | /user/hive/warehouse/schema_evolution.db/t1 |
+...
+
+-- Make T2 have the same data file as in T1, including 2
+-- unused columns and column order different than T2 expects.
+load data inpath '/user/hive/warehouse/schema_evolution.db/t1'
+  into table t2;
++----------------------------------------------------------+
+| summary                                                  |
++----------------------------------------------------------+
+| Loaded 1 file(s). Total files in destination location: 1 |
++----------------------------------------------------------+
+
+-- 'position' is the default setting.
+-- Impala cannot read the Parquet file if the column order does not match.
+set PARQUET_FALLBACK_SCHEMA_RESOLUTION=position;
+PARQUET_FALLBACK_SCHEMA_RESOLUTION set to position
+
+select * from t2;
+WARNINGS: 
+File 'schema_evolution.db/t2/45331705_data.0.parq'
+has an incompatible Parquet schema for column 'schema_evolution.t2.c4'.
+Column type: TIMESTAMP, Parquet schema: optional int32 c1 [i:0 d:1 r:0]
+
+File 'schema_evolution.db/t2/45331705_data.0.parq'
+has an incompatible Parquet schema for column 'schema_evolution.t2.c4'.
+Column type: TIMESTAMP, Parquet schema: optional int32 c1 [i:0 d:1 r:0]
+
+-- With the 'name' setting, Impala can read the Parquet data files
+-- despite mismatching column order.
+set PARQUET_FALLBACK_SCHEMA_RESOLUTION=name;
+PARQUET_FALLBACK_SCHEMA_RESOLUTION set to name
+
+select * from t2;
++-------------------------------+-------+
+| c4                            | c2    |
++-------------------------------+-------+
+| 2016-06-28 14:53:26.554369000 | true  |
+| 2016-06-29 14:53:26.554369000 | false |
++-------------------------------+-------+
+]]>
+</codeblock>
+
+      <note rev="IMPALA-3334" id="one_but_not_true">
+        In CDH 5.7.0 / Impala 2.5.0, only the value 1 enables the option, and the value
+        <codeph>true</codeph> is not recognized. This limitation is
+        tracked by the issue
+        <xref href="https://issues.cloudera.org/browse/IMPALA-3334" scope="external" format="html">IMPALA-3334</xref>,
+        which shows the releases where the problem is fixed.
+      </note>
+
+      <p rev="IMPALA-3732" id="avro_2gb_strings">
+        The Avro specification allows string values up to 2**64 bytes in length. 
+        Impala queries for Avro tables use 32-bit integers to hold string lengths.
+        In <keyword keyref="impala25_full"/> and higher, Impala truncates <codeph>CHAR</codeph>
+        and <codeph>VARCHAR</codeph> values in Avro tables to (2**31)-1 bytes.
+        If a query encounters a <codeph>STRING</codeph> value longer than (2**31)-1
+        bytes in an Avro table, the query fails. In earlier releases,
+        encountering such long values in an Avro table could cause a crash.
+      </p>
+
+      <p rev="2.6.0 IMPALA-3369" id="set_column_stats_example">
+        You specify a case-insensitive symbolic name for the kind of statistics:
+        <codeph>numDVs</codeph>, <codeph>numNulls</codeph>, <codeph>avgSize</codeph>, <codeph>maxSize</codeph>.
+        The key names and values are both quoted. This operation applies to an entire table,
+        not a specific partition. For example:
+<codeblock>
+create table t1 (x int, s string);
+insert into t1 values (1, 'one'), (2, 'two'), (2, 'deux');
+show column stats t1;
++--------+--------+------------------+--------+----------+----------+
+| Column | Type   | #Distinct Values | #Nulls | Max Size | Avg Size |
++--------+--------+------------------+--------+----------+----------+
+| x      | INT    | -1               | -1     | 4        | 4        |
+| s      | STRING | -1               | -1     | -1       | -1       |
++--------+--------+------------------+--------+----------+----------+
+alter table t1 set column stats x ('numDVs'='2','numNulls'='0');
+alter table t1 set column stats s ('numdvs'='3','maxsize'='4');
+show column stats t1;
++--------+--------+------------------+--------+----------+----------+
+| Column | Type   | #Distinct Values | #Nulls | Max Size | Avg Size |
++--------+--------+------------------+--------+----------+----------+
+| x      | INT    | 2                | 0      | 4        | 4        |
+| s      | STRING | 3                | -1     | 4        | -1       |
++--------+--------+------------------+--------+----------+----------+
+</codeblock>
+      </p>
+
+<codeblock id="set_numrows_example">create table analysis_data stored as parquet as select * from raw_data;
+Inserted 1000000000 rows in 181.98s
+compute stats analysis_data;
+insert into analysis_data select * from smaller_table_we_forgot_before;
+Inserted 1000000 rows in 15.32s
+-- Now there are 1001000000 rows. We can update this single data point in the stats.
+alter table analysis_data set tblproperties('numRows'='1001000000', 'STATS_GENERATED_VIA_STATS_TASK'='true');</codeblock>
+
+<codeblock id="set_numrows_partitioned_example">-- If the table originally contained 1 million rows, and we add another partition with 30 thousand rows,
+-- change the numRows property for the partition and the overall table.
+alter table partitioned_data partition(year=2009, month=4) set tblproperties ('numRows'='30000', 'STATS_GENERATED_VIA_STATS_TASK'='true');
+alter table partitioned_data set tblproperties ('numRows'='1030000', 'STATS_GENERATED_VIA_STATS_TASK'='true');</codeblock>
+
+      <p id="int_overflow_behavior">
+        Impala does not return column overflows as <codeph>NULL</codeph>, so that customers can distinguish
+        between <codeph>NULL</codeph> data and overflow conditions similar to how they do so with traditional
+        database systems. Impala returns the largest or smallest value in the range for the type. For example,
+        valid values for a <codeph>tinyint</codeph> range from -128 to 127. In Impala, a <codeph>tinyint</codeph>
+        with a value of -200 returns -128 rather than <codeph>NULL</codeph>. A <codeph>tinyint</codeph> with a
+        value of 200 returns 127.
+      </p>
+
+      <p rev="2.5.0" id="partition_key_optimization">
+        If you frequently run aggregate functions such as <codeph>MIN()</codeph>, <codeph>MAX()</codeph>, and
+        <codeph>COUNT(DISTINCT)</codeph> on partition key columns, consider enabling the <codeph>OPTIMIZE_PARTITION_KEY_SCANS</codeph>
+        query option, which optimizes such queries. This feature is available in <keyword keyref="impala25_full"/> and higher.
+        See <xref href="../topics/impala_optimize_partition_key_scans.xml"/>
+        for the kinds of queries that this option applies to, and slight differences in how partitions are
+        evaluated when this query option is enabled.
+      </p>
+
+      <p id="live_reporting_details">
+        The output from this query option is printed to standard error. The output is only displayed in interactive mode,
+        that is, not when the <codeph>-q</codeph> or <codeph>-f</codeph> options are used.
+      </p>
+
+      <p id="live_progress_live_summary_asciinema">
+        To see how the <codeph>LIVE_PROGRESS</codeph> and <codeph>LIVE_SUMMARY</codeph> query options
+        work in real time, see <xref href="https://asciinema.org/a/1rv7qippo0fe7h5k1b6k4nexk" scope="external" format="html">this animated demo</xref>.
+      </p>
+
+      <p rev="2.5.0" id="runtime_filter_mode_blurb">
+        Because the runtime filtering feature is enabled by default only for local processing,
+        the other filtering-related query options have the greatest effect when used in
+        combination with the setting <codeph>RUNTIME_FILTER_MODE=GLOBAL</codeph>.
+      </p>
+
+      <p rev="2.5.0" id="runtime_filtering_option_caveat">
+        Because the runtime filtering feature applies mainly to resource-intensive
+        and long-running queries, only adjust this query option when tuning long-running queries
+        involving some combination of large partitioned tables and joins involving large tables.
+      </p>
+
+      <p rev="2.3.0" id="impala_shell_progress_reports_compute_stats_caveat">
+        The <codeph>LIVE_PROGRESS</codeph> and <codeph>LIVE_SUMMARY</codeph> query options
+        currently do not produce any output during <codeph>COMPUTE STATS</codeph> operations.
+      </p>
+
+<!-- This is a shorter version of the similar 'caveat' text. This shorter one can be reused more easily in various places. -->
+      <p rev="2.3.0" id="impala_shell_progress_reports_shell_only_blurb">
+        The <codeph>LIVE_PROGRESS</codeph> and <codeph>LIVE_SUMMARY</codeph> query options only apply
+        inside the <cmdname>impala-shell</cmdname> interpreter. You cannot use them with the
+        <codeph>SET</codeph> statement from a JDBC or ODBC application.
+      </p>
+
+      <p id="impala_shell_progress_reports_shell_only_caveat">
+        Because the <codeph>LIVE_PROGRESS</codeph> and <codeph>LIVE_SUMMARY</codeph> query options
+        are available only within the <cmdname>impala-shell</cmdname> interpreter: 
+        <ul>
+          <li>
+            <p>
+              You cannot change these query options through the SQL <codeph>SET</codeph>
+              statement using the JDBC or ODBC interfaces. The <codeph>SET</codeph>
+              command in <cmdname>impala-shell</cmdname> recognizes these names as
+              shell-only options.
+            </p>
+          </li>
+          <li>
+            <p>
+              Be careful when using <cmdname>impala-shell</cmdname> on a pre-CDH 5.5
+              system to connect to Impala running on a CDH 5.5 or higher system.
+              The older <cmdname>impala-shell</cmdname> does not recognize these
+              query option names. Upgrade <cmdname>impala-shell</cmdname> on the
+              systems where you intend to use these query options.
+            </p>
+          </li>
+          <li>
+            <p>
+              Likewise, the <cmdname>impala-shell</cmdname> command relies on
+              some information only available in <keyword keyref="impala23_full"/> and higher
+              to prepare live progress reports and query summaries. The
+              <codeph>LIVE_PROGRESS</codeph> and <codeph>LIVE_SUMMARY</codeph> 
+              query options have no effect when <cmdname>impala-shell</cmdname> connects
+              to a cluster running an older version of Impala.
+            </p>
+          </li>
+        </ul>
+      </p>
+
+<!-- Same example used in both CREATE DATABASE and DROP DATABASE. -->
+<codeblock id="create_drop_db_example">create database first_db;
+use first_db;
+create table t1 (x int);
+
+create database second_db;
+use second_db;
+-- Each database has its own namespace for tables.
+-- You can reuse the same table names in each database.
+create table t1 (s string);
+
+create database temp;
+
+-- You can either USE a database after creating it,
+-- or qualify all references to the table name with the name of the database.
+-- Here, tables T2 and T3 are both created in the TEMP database.
+
+create table temp.t2 (x int, y int);
+use database temp;
+create table t3 (s string);
+
+-- You cannot drop a database while it is selected by the USE statement.
+drop database temp;
+<i>ERROR: AnalysisException: Cannot drop current default database: temp</i>
+
+-- The always-available database 'default' is a convenient one to USE
+-- before dropping a database you created.
+use default;
+
+-- Before dropping a database, first drop all the tables inside it,
+<ph rev="2.3.0">-- or in <keyword keyref="impala23_full"/> and higher use the CASCADE clause.</ph>
+drop database temp;
+ERROR: ImpalaRuntimeException: Error making 'dropDatabase' RPC to Hive Metastore: 
+CAUSED BY: InvalidOperationException: Database temp is not empty
+show tables in temp;
++------+
+| name |
++------+
+| t3   |
++------+
+
+<ph rev="2.3.0">-- <keyword keyref="impala23_full"/> and higher:</ph>
+<ph rev="2.3.0">drop database temp cascade;</ph>
+
+-- CDH 5.4 and lower:
+drop table temp.t3;
+drop database temp;
+</codeblock>
+
+      <p id="cast_convenience_fn_example">
+        This example shows how to use the <codeph>castto*()</codeph> functions as an equivalent
+        to <codeph>CAST(<varname>value</varname> AS <varname>type</varname>)</codeph> expressions.
+      </p>
+
+      <p id="cast_convenience_fn_usage"><b>Usage notes:</b>
+        A convenience function to skip the SQL <codeph>CAST <varname>value</varname> AS <varname>type</varname></codeph> syntax,
+        for example when programmatically generating SQL statements where a regular function call might be easier to construct.
+      </p>
+
+      <p rev="2.3.0" id="current_timezone_tip">
+        To determine the time zone of the server you are connected to, in CDH 5.5 / Impala 2.3 and
+        higher you can call the <codeph>timeofday()</codeph> function, which includes the time zone
+        specifier in its return value. Remember that with cloud computing, the server you interact
+        with might be in a different time zone than you are, or different sessions might connect to
+        servers in different time zones, or a cluster might include servers in more than one time zone.
+      </p>
+
+      <p rev="2.2.0" id="timezone_conversion_caveat">
+        The way this function deals with time zones when converting to or from <codeph>TIMESTAMP</codeph>
+        values is affected by the <codeph>-use_local_tz_for_unix_timestamp_conversions</codeph> startup flag for the
+        <cmdname>impalad</cmdname> daemon. See <xref href="../topics/impala_timestamp.xml#timestamp"/> for details about
+        how Impala handles time zone considerations for the <codeph>TIMESTAMP</codeph> data type.
+      </p>
+
+      <p rev="2.6.0 CDH-39913 IMPALA-3558" id="s3_drop_table_purge">
+        For best compatibility with the S3 write support in CDH 5.8 / Impala 2.6
+        and higher:
+        <ul>
+        <li>Use native Hadoop techniques to create data files in S3 for querying through Impala.</li>
+        <li>Use the <codeph>PURGE</codeph> clause of <codeph>DROP TABLE</codeph> when dropping internal (managed) tables.</li>
+        </ul>
+        By default, when you drop an internal (managed) table, the data files are
+        moved to the HDFS trashcan. This operation is expensive for tables that
+        reside on the Amazon S3 filesystem. Therefore, for S3 tables, prefer to use
+        <codeph>DROP TABLE <varname>table_name</varname> PURGE</codeph> rather than the default <codeph>DROP TABLE</codeph> statement.
+        The <codeph>PURGE</codeph> clause makes Impala delete the data files immediately,
+        skipping the HDFS trashcan.
+        For the <codeph>PURGE</codeph> clause to work effectively, you must originally create the
+        data files on S3 using one of the tools from the Hadoop ecosystem, such as
+        <codeph>hadoop fs -cp</codeph>, or <codeph>INSERT</codeph> in Impala or Hive.
+      </p>
+
+      <p rev="2.6.0 CDH-39913 IMPALA-1878" id="s3_dml_performance">
+        Because of differences between S3 and traditional filesystems, DML operations
+        for S3 tables can take longer than for tables on HDFS. For example, both the
+        <codeph>LOAD DATA</codeph> statement and the final stage of the <codeph>INSERT</codeph>
+        and <codeph>CREATE TABLE AS SELECT</codeph> statements involve moving files from one directory
+        to another. (In the case of <codeph>INSERT</codeph> and <codeph>CREATE TABLE AS SELECT</codeph>,
+        the files are moved from a temporary staging directory to the final destination directory.)
+        Because S3 does not support a <q>rename</q> operation for existing objects, in these cases Impala
+        actually copies the data files from one location to another and then removes the original files.
+        In CDH 5.8 / Impala 2.6, the <codeph>S3_SKIP_INSERT_STAGING</codeph> query option provides a way
+        to speed up <codeph>INSERT</codeph> statements for S3 tables and partitions, with the tradeoff
+        that a problem during statement execution could leave data in an inconsistent state.
+        It does not apply to <codeph>INSERT OVERWRITE</codeph> or <codeph>LOAD DATA</codeph> statements.
+        See <xref href="../topics/impala_s3_skip_insert_staging.xml#s3_skip_insert_staging"/> for details.
+      </p>
+
+      <p rev="2.6.0 CDH-40329 IMPALA-3453" id="s3_block_splitting">
+        In <keyword keyref="impala26_full"/> and higher, Impala queries are optimized for files stored in Amazon S3.
+        For Impala tables that use the file formats Parquet, RCFile, SequenceFile,
+        Avro, and uncompressed text, the setting <codeph>fs.s3a.block.size</codeph>
+        in the <filepath>core-site.xml</filepath> configuration file determines
+        how Impala divides the I/O work of reading the data files. This configuration
+        setting is specified in bytes. By default, this
+        value is 33554432 (32 MB), meaning that Impala parallelizes S3 read operations on the files
+        as if they were made up of 32 MB blocks. For example, if your S3 queries primarily access
+        Parquet files written by MapReduce or Hive, increase <codeph>fs.s3a.block.size</codeph>
+        to 134217728 (128 MB) to match the row group size of those files. If most S3 queries involve
+        Parquet files written by Impala, increase <codeph>fs.s3a.block.size</codeph>
+        to 268435456 (256 MB) to match the row group size produced by Impala.
+      </p>
+
+      <note rev="2.6.0 CDH-39913 IMPALA-1878" id="s3_production" type="important">
+        <p>
+          In <keyword keyref="impala26_full"/> and higher, Impala supports both queries (<codeph>SELECT</codeph>)
+          and DML (<codeph>INSERT</codeph>, <codeph>LOAD DATA</codeph>, <codeph>CREATE TABLE AS SELECT</codeph>)
+          for data residing on Amazon S3. With the inclusion of write support,
+          <!-- and configuration settings for more secure S3 key management, -->
+          the Impala support for S3 is now considered ready for production use.
+        </p>
+      </note>
+
+      <note rev="2.2.0" id="s3_caveat" type="important">
+        <p> Impala query support for Amazon S3 is included in CDH 5.4.0, but is
+          not currently supported or recommended for production use. To try this
+          feature, use it in a test environment until Cloudera resolves
+          currently existing issues and limitations to make it ready for
+          production use. </p>
+      </note>
+
+      <p rev="2.6.0 CDH-39913 IMPALA-1878" id="s3_ddl">
+        In <keyword keyref="impala26_full"/> and higher, Impala DDL statements such as
+        <codeph>CREATE DATABASE</codeph>, <codeph>CREATE TABLE</codeph>, <codeph>DROP DATABASE CASCADE</codeph>,
+        <codeph>DROP TABLE</codeph>, and <codeph>ALTER TABLE [ADD|DROP] PARTITION</codeph> can create or remove folders
+        as needed in the Amazon S3 system. Prior to CDH 5.8 / Impala 2.6, you had to create folders yourself and point
+        Impala database, tables, or partitions at them, and manually remove folders when no longer needed.
+        See <xref href="../topics/impala_s3.xml#s3"/> for details about reading and writing S3 data with Impala.
+      </p>
+
+      <p rev="2.6.0 CDH-39913 IMPALA-1878" id="s3_dml">
+        In <keyword keyref="impala26_full"/> and higher, the Impala DML statements (<codeph>INSERT</codeph>, <codeph>LOAD DATA</codeph>,
+        and <codeph>CREATE TABLE AS SELECT</codeph>) can write data into a table or partition that resides in the
+        Amazon Simple Storage Service (S3).
+        The syntax of the DML statements is the same as for any other tables, because the S3 location for tables and
+        partitions is specified by an <codeph>s3a://</codeph> prefix in the
+        <codeph>LOCATION</codeph> attribute of
+        <codeph>CREATE TABLE</codeph> or <codeph>ALTER TABLE</codeph> statements.
+        If you bring data into S3 using the normal S3 transfer mechanisms instead of Impala DML statements,
+        issue a <codeph>REFRESH</codeph> statement for the table before using Impala to query the S3 data.
+      </p>
+
+        <!-- Formerly part of s3_dml element. Moved out to avoid a circular link in the S3 topic itelf. -->
+        <!-- See <xref href="../topics/impala_s3.xml#s3"/> for details about reading and writing S3 data with Impala. -->
+
+      <p rev="2.2.0" id="s3_metadata">
+        The <codeph>REFRESH</codeph> and <codeph>INVALIDATE METADATA</codeph> statements also cache metadata
+        for tables where the data resides in the Amazon Simple Storage Service (S3).
+        In particular, issue a <codeph>REFRESH</codeph> for a table after adding or removing files
+        in the associated S3 data directory.
+        See <xref href="../topics/impala_s3.xml#s3"/> for details about working with S3 tables.
+      </p>
+
+      <p id="y2k38" rev="2.2.0">
+        In Impala 2.2.0 and higher, built-in functions that accept or return integers representing <codeph>TIMESTAMP</codeph> values
+        use the <codeph>BIGINT</codeph> type for parameters and return values, rather than <codeph>INT</codeph>.
+        This change lets the date and time functions avoid an overflow error that would otherwise occur
+        on January 19th, 2038 (known as the
+        <xref href="http://en.wikipedia.org/wiki/Year_2038_problem" scope="external" format="html"><q>Year 2038 problem</q> or <q>Y2K38 problem</q></xref>).
+        This change affects the <codeph>from_unixtime()</codeph> and <codeph>unix_timestamp()</codeph> functions.
+        You might need to change application code that interacts with these functions, change the types of
+        columns that store the return values, or add <codeph>CAST()</codeph> calls to SQL statements that
+        call these functions.
+      </p>
+
+      <p id="timestamp_conversions">
+        Impala automatically converts <codeph>STRING</codeph> literals of the correct format into
+        <codeph>TIMESTAMP</codeph> values. Timestamp values are accepted in the format
+        <codeph>"yyyy-MM-dd HH:mm:ss.SSSSSS"</codeph>, and can consist of just the date, or just the time, with or
+        without the fractional second portion. For example, you can specify <codeph>TIMESTAMP</codeph> values such as
+        <codeph>'1966-07-30'</codeph>, <codeph>'08:30:00'</codeph>, or <codeph>'1985-09-25 17:45:30.005'</codeph>.
+        <ph conref="../shared/impala_common.xml#common/cast_int_to_timestamp"/>
+      </p>
+
+
+      <p>
+        <ph id="cast_int_to_timestamp">Casting an integer or floating-point value <codeph>N</codeph> to
+        <codeph>TIMESTAMP</codeph> produces a value that is <codeph>N</codeph> seconds past the start of the epoch
+        date (January 1, 1970). By default, the result value represents a date and time in the UTC time zone.
+        If the setting <codeph>-use_local_tz_for_unix_timestamp_conversions=true</codeph> is in effect,
+        the resulting <codeph>TIMESTAMP</codeph> represents a date and time in the local time zone.</ph>
+      </p>
+
+      <p id="redaction_yes" rev="2.2.0">
+        If these statements in your environment contain sensitive literal values such as credit card numbers or tax
+        identifiers, Impala can redact this sensitive information when displaying the statements in log files and
+        other administrative contexts. See
+        <xref audience="integrated" href="../topics/sg_redaction.xml#log_redact"/><xref audience="standalone" href="http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/sg_redaction.html" scope="external" format="html"/>
+        for details.
+      </p>
+
+      <p id="incremental_partition_spec">
+        The <codeph>PARTITION</codeph> clause is only allowed in combination with the <codeph>INCREMENTAL</codeph>
+        clause. It is optional for <codeph>COMPUTE INCREMENTAL STATS</codeph>, and required for <codeph>DROP
+        INCREMENTAL STATS</codeph>. Whenever you specify partitions through the <codeph>PARTITION
+        (<varname>partition_spec</varname>)</codeph> clause in a <codeph>COMPUTE INCREMENTAL STATS</codeph> or
+        <codeph>DROP INCREMENTAL STATS</codeph> statement, you must include all the partitioning columns in the
+        specification, and specify constant values for all the partition key columns.
+      </p>
+
+      <p id="udf_persistence_restriction" rev="2.5.0 IMPALA-1748">
+        In <keyword keyref="impala25_full"/> and higher, Impala UDFs and UDAs written in C++ are persisted in the metastore database.
+        Java UDFs are also persisted, if they were created with the new <codeph>CREATE FUNCTION</codeph> syntax for Java UDFs,
+        where the Java function argument and return types are omitted.
+        Java-based UDFs created with the old <codeph>CREATE FUNCTION</codeph> syntax do not persist across restarts
+        because they are held in the memory of the <cmdname>catalogd</cmdname> daemon.
+        Until you re-create such Java UDFs using the new <codeph>CREATE FUNCTION</codeph> syntax,
+        you must reload those Java-based UDFs by running the original <codeph>CREATE FUNCTION</codeph> statements again each time
+        you restart the <cmdname>catalogd</cmdname> daemon.
+        Prior to <keyword keyref="impala25_full"/> the requirement to reload functions after a restart applied to both C++ and Java functions.
+      </p>
+
+      <p id="current_user_caveat" rev="CDH-36552">
+        The Hive <codeph>current_user()</codeph> function cannot be
+        called from a Java UDF through Impala.
+      </p>
+
+      <note id="add_partition_set_location">
+        If you are creating a partition for the first time and specifying its location, for maximum efficiency, use
+        a single <codeph>ALTER TABLE</codeph> statement including both the <codeph>ADD PARTITION</codeph> and
+        <codeph>LOCATION</codeph> clauses, rather than separate statements with <codeph>ADD PARTITION</codeph> and
+        <codeph>SET LOCATION</codeph> clauses.
+      </note>
+
+      <p id="insert_hidden_work_directory">
+        The <codeph>INSERT</codeph> statement has always left behind a hidden work directory inside the data
+        directory of the table. Formerly, this hidden work directory was named
+        <filepath>.impala_insert_staging</filepath> . In Impala 2.0.1 and later, this directory name is changed to
+        <filepath>_impala_insert_staging</filepath> . (While HDFS tools are expected to treat names beginning
+        either with underscore and dot as hidden, in practice names beginning with an underscore are more widely
+        supported.) If you have any scripts, cleanup jobs, and so on that rely on the name of this work directory,
+        adjust them to use the new name.
+      </p>
+
+      <p id="check_internal_external_table">
+        To see whether a table is internal or external, and its associated HDFS location, issue the statement
+        <codeph>DESCRIBE FORMATTED <varname>table_name</varname></codeph>. The <codeph>Table Type</codeph> field
+        displays <codeph>MANAGED_TABLE</codeph> for internal tables and <codeph>EXTERNAL_TABLE</codeph> for
+        external tables. The <codeph>Location</codeph> field displays the path of the table directory as an HDFS
+        URI.
+      </p>
+
+      <p id="switch_internal_external_table">
+        You can switch a table from internal to external, or from external to internal, by using the <codeph>ALTER
+        TABLE</codeph> statement:
+<codeblock xml:space="preserve">
+-- Switch a table from internal to external.
+ALTER TABLE <varname>table_name</varname> SET TBLPROPERTIES('EXTERNAL'='TRUE');
+
+-- Switch a table from external to internal.
+ALTER TABLE <varname>table_name</varname> SET TBLPROPERTIES('EXTERNAL'='FALSE');
+</codeblock>
+      </p>
+
+<!-- The data to show sensible output from these queries is in the TPC-DS schema 'CUSTOMER' table.
+     If you want to show real output, add a LIMIT 5 or similar clause to each query to avoid
+     too-long output. -->
+
+<codeblock id="regexp_rlike_examples" xml:space="preserve">-- Find all customers whose first name starts with 'J', followed by 0 or more of any character.
+select c_first_name, c_last_name from customer where c_first_name regexp '^J.*';
+select c_first_name, c_last_name from customer where c_first_name rlike '^J.*';
+
+-- Find 'Macdonald', where the first 'a' is optional and the 'D' can be upper- or lowercase.
+-- The ^...$ are required, to match the start and end of the value.
+select c_first_name, c_last_name from customer where c_last_name regexp '^Ma?c[Dd]onald$';
+select c_first_name, c_last_name from customer where c_last_name rlike '^Ma?c[Dd]onald$';
+
+-- Match multiple character sequences, either 'Mac' or 'Mc'.
+select c_first_name, c_last_name from customer where c_last_name regexp '^(Mac|Mc)donald$';
+select c_first_name, c_last_name from customer where c_last_name rlike '^(Mac|Mc)donald$';
+
+-- Find names starting with 'S', then one or more vowels, then 'r', then any other characters.
+-- Matches 'Searcy', 'Sorenson', 'Sauer'.
+select c_first_name, c_last_name from customer where c_last_name regexp '^S[aeiou]+r.*$';
+select c_first_name, c_last_name from customer where c_last_name rlike '^S[aeiou]+r.*$';
+
+-- Find names that end with 2 or more vowels: letters from the set a,e,i,o,u.
+select c_first_name, c_last_name from customer where c_last_name regexp '.*[aeiou]{2,}$';
+select c_first_name, c_last_name from customer where c_last_name rlike '.*[aeiou]{2,}$';
+
+-- You can use letter ranges in the [] blocks, for example to find names starting with A, B, or C.
+select c_first_name, c_last_name from customer where c_last_name regexp '^[A-C].*';
+select c_first_name, c_last_name from customer where c_last_name rlike '^[A-C].*';
+
+-- If you are not sure about case, leading/trailing spaces, and so on, you can process the
+-- column using string functions first.
+select c_first_name, c_last_name from customer where lower(trim(c_last_name)) regexp '^de.*';
+select c_first_name, c_last_name from customer where lower(trim(c_last_name)) rlike '^de.*';
+</codeblock>
+
+      <p id="case_insensitive_comparisons_tip" rev="2.5.0 IMPALA-1787">
+        In <keyword keyref="impala25_full"/> and higher, you can simplify queries that
+        use many <codeph>UPPER()</codeph> and <codeph>LOWER()</codeph> calls
+        to do case-insensitive comparisons, by using the <codeph>ILIKE</codeph>
+        or <codeph>IREGEXP</codeph> operators instead. See
+        <xref href="../topics/impala_operators.xml#ilike"/> and
+        <xref href="../topics/impala_operators.xml#iregexp"/> for details.
+      </p>
+
+      <p id="show_security">
+        When authorization is enabled, the output of the <codeph>SHOW</codeph> statement is limited to those
+        objects for which you have some privilege. There might be other database, tables, and so on, but their
+        names are concealed. If you believe an object exists but you cannot see it in the <codeph>SHOW</codeph>
+        output, check with the system administrator if you need to be granted a new privilege for that object. See
+        <xref href="../topics/impala_authorization.xml#authorization"/> for how to set up authorization and add
+        privileges for specific kinds of objects.
+      </p>
+
+      <p id="infinity_and_nan" rev="IMPALA-3267">
+        Infinity and NaN can be specified in text data files as <codeph>inf</codeph> and <codeph>nan</codeph>
+        respectively, and Impala interprets them as these special values. They can also be produced by certain
+        arithmetic expressions; for example, <codeph>pow(-1, 0.5)</codeph> returns <codeph>Infinity</codeph> and
+        <codeph>1/0</codeph> returns <codeph>NaN</codeph>. Or you can cast the literal values, such as <codeph>CAST('nan' AS
+        DOUBLE)</codeph> or <codeph>CAST('inf' AS DOUBLE)</codeph>.
+      </p>
+
+      <p rev="2.0.0" id="user_kerberized">
+        In Impala 2.0 and later, <codeph>user()</codeph> returns the full Kerberos principal string, such as
+        <codeph>user@example.com</codeph>, in a Kerberized environment.
+      </p>
+
+      <ul>
+        <li id="grant_revoke_single">
+          Currently, each Impala <codeph>GRANT</codeph> or <codeph>REVOKE</codeph> statement can only grant or
+          revoke a single privilege to or from a single role.
+        </li>
+      </ul>
+
+      <p id="blobs_are_strings">
+        All data in <codeph>CHAR</codeph> and <codeph>VARCHAR</codeph> columns must be in a character encoding that
+        is compatible with UTF-8. If you have binary data from another database system (that is, a BLOB type), use
+        a <codeph>STRING</codeph> column to hold it.
+      </p>
+
+<!-- The codeblock is nested inside this paragraph, so the intro text
+     and the code get conref'ed as a unit. -->
+
+      <p id="create_drop_view_examples">
+        The following example creates a series of views and then drops them. These examples illustrate how views
+        are associated with a particular database, and both the view definitions and the view names for
+        <codeph>CREATE VIEW</codeph> and <codeph>DROP VIEW</codeph> can refer to a view in the current database or
+        a fully qualified view name.
+<codeblock xml:space="preserve">
+-- Create and drop a view in the current database.
+CREATE VIEW few_rows_from_t1 AS SELECT * FROM t1 LIMIT 10;
+DROP VIEW few_rows_from_t1;
+
+-- Create and drop a view referencing a table in a different database.
+CREATE VIEW table_from_other_db AS SELECT x FROM db1.foo WHERE x IS NOT NULL;
+DROP VIEW table_from_other_db;
+
+USE db1;
+-- Create a view in a different database.
+CREATE VIEW db2.v1 AS SELECT * FROM db2.foo;
+-- Switch into the other database and drop the view.
+USE db2;
+DROP VIEW v1;
+
+USE db1;
+-- Create a view in a different database.
+CREATE VIEW db2.v1 AS SELECT * FROM db2.foo;
+-- Drop a view in the other database.
+DROP VIEW db2.v1;
+</codeblock>
+      </p>
+
+      <p id="char_varchar_cast_from_string">
+        For <codeph>INSERT</codeph> operations into <codeph>CHAR</codeph> or <codeph>VARCHAR</codeph> columns, you
+        must cast all <codeph>STRING</codeph> literals or expressions returning <codeph>STRING</codeph> to to a
+        <codeph>CHAR</codeph> or <codeph>VARCHAR</codeph> type with the appropriate length.
+      </p>
+
+      <p rev="2.0.0" id="subquery_no_limit">
+        Correlated subqueries used in <codeph>EXISTS</codeph> and <codeph>IN</codeph> operators cannot include a
+        <codeph>LIMIT</codeph> clause.
+      </p>
+
+      <p id="avro_no_timestamp">
+        Currently, Avro tables cannot contain <codeph>TIMESTAMP</codeph> columns. If you need to store date and
+        time values in Avro tables, as a workaround you can use a <codeph>STRING</codeph> representation of the
+        values, convert the values to <codeph>BIGINT</codeph> with the <codeph>UNIX_TIMESTAMP()</codeph> function,
+        or create separate numeric columns for individual date and time fields using the <codeph>EXTRACT()</codeph>
+        function.
+      </p>
+
+      <p id="zero_length_strings">
+        <b>Zero-length strings:</b> For purposes of clauses such as <codeph>DISTINCT</codeph> and <codeph>GROUP
+        BY</codeph>, Impala considers zero-length strings (<codeph>""</codeph>), <codeph>NULL</codeph>, and space
+        to all be different values.
+      </p>
+
+      <p rev="2.5.0 IMPALA-3054" id="spill_to_disk_vs_dynamic_partition_pruning">
+        When the spill-to-disk feature is activated for a join node within a query, Impala does not
+        produce any runtime filters for that join operation on that host. Other join nodes within
+        the query are not affected.
+      </p>
+
+<codeblock id="simple_dpp_example">
+create table yy (s string) partitioned by (year int) stored as parquet;
+insert into yy partition (year) values ('1999', 1999), ('2000', 2000),
+  ('2001', 2001), ('2010',2010);
+compute stats yy;
+
+create table yy2 (s string) partitioned by (year int) stored as parquet;
+insert into yy2 partition (year) values ('1999', 1999), ('2000', 2000),
+  ('2001', 2001);
+compute stats yy2;
+
+-- The query reads an unknown number of partitions, whose key values are only
+-- known at run time. The 'runtime filters' lines show how the information about
+-- the partitions is calculated in query fragment 02, and then used in query
+-- fragment 00 to decide which partitions to skip.
+explain select s from yy2 where year in (select year from yy where year between 2000 and 2005);
++----------------------------------------------------------+
+| Explain String                                           |
++----------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=16.00MB VCores=2 |
+|                                                          |
+| 04:EXCHANGE [UNPARTITIONED]                              |
+| |                                                        |
+| 02:HASH JOIN [LEFT SEMI JOIN, BROADCAST]                 |
+| |  hash predicates: year = year                          |
+| |  <b>runtime filters: RF000 &lt;- year</b>                        |
+| |                                                        |
+| |--03:EXCHANGE [BROADCAST]                               |
+| |  |                                                     |
+| |  01:SCAN HDFS [dpp.yy]                                 |
+| |     partitions=2/4 files=2 size=468B                   |
+| |                                                        |
+| 00:SCAN HDFS [dpp.yy2]                                   |
+|    partitions=2/3 files=2 size=468B                      |
+|    <b>runtime filters: RF000 -> year</b>                        |
++----------------------------------------------------------+
+</codeblock>
+      <p id="order_by_scratch_dir">
+        By default, intermediate files used during large sort, join, aggregation, or analytic function operations
+        are stored in the directory <filepath>/tmp/impala-scratch</filepath> . These files are removed when the
+        operation finishes. (Multiple concurrent queries can perform operations that use the <q>spill to disk</q>
+        technique, without any name conflicts for these temporary files.) You can specify a different location by
+        starting the <cmdname>impalad</cmdname> daemon with the
+        <codeph>--scratch_dirs="<varname>path_to_directory</varname>"</codeph> configuration option or the
+        equivalent configuration option in the Cloudera Manager user interface. You can specify a single directory,
+        or a comma-separated list of directories. The scratch directories must be on the local filesystem, not in
+        HDFS. You might specify different directory paths for different hosts, depending on the capacity and speed
+        of the available storage devices. In CDH 5.5 / Impala 2.3 or higher, Impala successfully starts (with a warning
+        written to the log) if it cannot create or read and write files in one of the scratch directories.
+        If there is less than 1 GB free on the filesystem where that directory resides, Impala still runs, but writes a
+        warning message to its log.  If Impala encounters an error reading or writing files in a scratch directory during
+        a query, Impala logs the error and the query fails.
+      </p>
+
+      <p id="order_by_view_restriction">
+        An <codeph>ORDER BY</codeph> clause without an additional <codeph>LIMIT</codeph> clause is ignored in any
+        view definition. If you need to sort the entire result set from a view, use an <codeph>ORDER BY</codeph>
+        clause in the <codeph>SELECT</codeph> statement that queries the view. You can still make a simple <q>top
+        10</q> report by combining the <codeph>ORDER BY</codeph> and <codeph>LIMIT</codeph> clauses in the same
+        view definition:
+<codeblock xml:space="preserve">[localhost:21000] &gt; create table unsorted (x bigint);
+[localhost:21000] &gt; insert into unsorted values (1), (9), (3), (7), (5), (8), (4), (6), (2);
+[localhost:21000] &gt; create view sorted_view as select x from unsorted order by x;
+[localhost:21000] &gt; select x from sorted_view; -- ORDER BY clause in view has no effect.
++---+
+| x |
++---+
+| 1 |
+| 9 |
+| 3 |
+| 7 |
+| 5 |
+| 8 |
+| 4 |
+| 6 |
+| 2 |
++---+
+[localhost:21000] &gt; select x from sorted_view order by x; -- View query requires ORDER BY at outermost level.
++---+
+| x |
++---+
+| 1 |
+| 2 |
+| 3 |
+| 4 |
+| 5 |
+| 6 |
+| 7 |
+| 8 |
+| 9 |
++---+
+[localhost:21000] &gt; create view top_3_view as select x from unsorted order by x limit 3;
+[localhost:21000] &gt; select x from top_3_view; -- ORDER BY and LIMIT together in view definition are preserved.
++---+
+| x |
++---+
+| 1 |
+| 2 |
+| 3 |
++---+
+</codeblock>
+      </p>
+
+      <p id="precision_scale_example">
+        The following examples demonstrate how to check the precision and scale of numeric literals or other
+        numeric expressions. Impala represents numeric literals in the smallest appropriate type. 5 is a
+        <codeph>TINYINT</codeph> value, which ranges from -128 to 127, therefore 3 decimal digits are needed to
+        represent the entire range, and because it is an integer value there are no fractional digits. 1.333 is
+        interpreted as a <codeph>DECIMAL</codeph> value, with 4 digits total and 3 digits after the decimal point.
+<codeblock xml:space="preserve">[localhost:21000] &gt; select precision(5), scale(5);
++--------------+----------+
+| precision(5) | scale(5) |
++--------------+----------+
+| 3            | 0        |
++--------------+----------+
+[localhost:21000] &gt; select precision(1.333), scale(1.333);
++------------------+--------------+
+| precision(1.333) | scale(1.333) |
++------------------+--------------+
+| 4                | 3            |
++------------------+--------------+
+[localhost:21000] &gt; with t1 as
+  ( select cast(12.34 as decimal(20,2)) x union select cast(1 as decimal(8,6)) x )
+  select precision(x), scale(x) from t1 limit 1;
++--------------+----------+
+| precision(x) | scale(x) |
++--------------+----------+
+| 24           | 6        |
++--------------+----------+
+</codeblock>
+      </p>
+
+<!-- These 'type_' entries are for query options, where the type doesn't match up exactly with an Impala data type. -->
+
+      <p id="type_boolean">
+        <b>Type:</b> Boolean; recognized values are 1 and 0, or <codeph>true</codeph> and <codeph>false</codeph>;
+        any other value interpreted as <codeph>false</codeph>
+      </p>
+
+      <p id="type_string">
+        <b>Type:</b> string
+      </p>
+
+      <p id="type_integer">
+        <b>Type:</b> integer
+      </p>
+
+      <p id="default_false">
+        <b>Default:</b> <codeph>false</codeph>
+      </p>
+
+      <p id="default_false_0">
+        <b>Default:</b> <codeph>false</codeph> (shown as 0 in output of <codeph>SET</codeph> statement)
+      </p>
+
+      <p id="default_true_1">
+        <b>Default:</b> <codeph>true</codeph> (shown as 1 in output of <codeph>SET</codeph> statement)
+      </p>
+
+      <p id="odd_return_type_string">
+        Currently, the return value is always a <codeph>STRING</codeph>. The return type is subject to change in
+        future releases. Always use <codeph>CAST()</codeph> to convert the result to whichever data type is
+        appropriate for your computations.
+      </p>
+
+      <p rev="2.0.0" id="former_odd_return_type_string">
+        <b>Return type:</b> <codeph>DOUBLE</codeph> in Impala 2.0 and higher; <codeph>STRING</codeph> in earlier
+        releases
+      </p>
+
+      <p id="for_compatibility_only">
+        <b>Usage notes:</b> Primarily for compatibility with code containing industry extensions to SQL.
+      </p>
+
+      <p id="return_type_boolean">
+        <b>Return type:</b> <codeph>BOOLEAN</codeph>
+      </p>
+
+      <p id="return_type_double">
+        <b>Return type:</b> <codeph>DOUBLE</codeph>
+      </p>
+
+      <p id="return_type_same">
+        <b>Return type:</b> Same as the input value
+      </p>
+
+      <p id="return_type_same_except_string">
+        <b>Return type:</b> Same as the input value, except for <codeph>CHAR</codeph> and <codeph>VARCHAR</codeph>
+        arguments which produce a <codeph>STRING</codeph> result
+      </p>
+
+      <p id="builtins_db">
+        Impala includes another predefined database, <codeph>_impala_builtins</codeph>, that serves as the location
+        for the <xref href="../topics/impala_functions.xml#builtins">built-in functions</xref>. To see the built-in
+        functions, use a statement like the following:
+<codeblock xml:space="preserve">show functions in _impala_builtins;
+show functions in _impala_builtins like '*<varname>substring</varname>*';
+</codeblock>
+      </p>
+
+      <p id="sum_double">
+        Due to the way arithmetic on <codeph>FLOAT</codeph> and <codeph>DOUBLE</codeph> columns uses
+        high-performance hardware instructions, and distributed queries can perform these operations in different
+        order for each query, results can vary slightly for aggregate function calls such as <codeph>SUM()</codeph>
+        and <codeph>AVG()</codeph> for <codeph>FLOAT</codeph> and <codeph>DOUBLE</codeph> columns, particularly on
+        large data sets where millions or billions of values are summed or averaged. For perfect consistency and
+        repeatability, use the <codeph>DECIMAL</codeph> data type for such operations instead of
+        <codeph>FLOAT</codeph> or <codeph>DOUBLE</codeph>.
+      </p>
+
+      <p id="float_double_decimal_caveat">
+        The inability to exactly represent certain floating-point values means that
+        <codeph>DECIMAL</codeph> is sometimes a better choice than <codeph>DOUBLE</codeph>
+        or <codeph>FLOAT</codeph> when precision is critical, particularly when
+        transferring data from other database systems that use different representations
+        or file formats.
+      </p>
+
+      <p rev="1.4.0" id="decimal_no_stats">
+        Currently, the <codeph>COMPUTE STATS</codeph> statement under CDH 4 does not store any statistics for
+        <codeph>DECIMAL</codeph> columns. When Impala runs under CDH 5, which has better support for
+        <codeph>DECIMAL</codeph> in the metastore database, <codeph>COMPUTE STATS</codeph> does collect statistics
+        for <codeph>DECIMAL</codeph> columns and Impala uses the statistics to optimize query performance.
+      </p>
+
+      <p rev="CDH-35866" id="hive_column_stats_caveat">
+        If you run the Hive statement <codeph>ANALYZE TABLE COMPUTE STATISTICS FOR COLUMNS</codeph>, 
+        Impala can only use the resulting column statistics if the table is unpartitioned.
+        Impala cannot use Hive-generated column statistics for a partitioned table.
+      </p>
+
+      <p id="datetime_function_chaining">
+        <codeph>unix_timestamp()</codeph> and <codeph>from_unixtime()</codeph> are often used in combination to
+        convert a <codeph>TIMESTAMP</codeph> value into a particular string format. For example:
+<codeblock xml:space="preserve">select from_unixtime(unix_timestamp(now() + interval 3 days),
+  'yyyy/MM/dd HH:mm') as yyyy_mm_dd_hh_mm;
++------------------+
+| yyyy_mm_dd_hh_mm |
++------------------+
+| 2016/06/03 11:38 |
++------------------+
+</codeblock>
+      </p>
+
+      <p rev="1.4.0 obwl" id="insert_sort_blurb">
+        <b>Sorting considerations:</b> Although you can specify an <codeph>ORDER BY</codeph> clause in an
+        <codeph>INSERT ... SELECT</codeph> statement, any <codeph>ORDER BY</codeph> clause is ignored and the
+        results are not necessarily sorted. An <codeph>INSERT ... SELECT</codeph> operation potentially creates
+        many different data files, prepared on different data nodes, and therefore the notion of the data being
+        stored in sorted order is impractical.
+      </p>
+
+      <p rev="1.4.0" id="create_table_like_view">
+        Prior to Impala 1.4.0, it was not possible to use the <codeph>CREATE TABLE LIKE
+        <varname>view_name</varname></codeph> syntax. In Impala 1.4.0 and higher, you can create a table with the
+        same column definitions as a view using the <codeph>CREATE TABLE LIKE</codeph> technique. Although
+        <codeph>CREATE TABLE LIKE</codeph> normally inherits the file format of the original table, a view has no
+        underlying file format, so <codeph>CREATE TABLE LIKE <varname>view_name</varname></codeph> produces a text
+        table by default. To specify a different file format, include a <codeph>STORED AS
+        <varname>file_format</varname></codeph> clause at the end of the <codeph>CREATE TABLE LIKE</codeph>
+        statement.
+      </p>
+
+      <note rev="1.4.0" id="compute_stats_nulls"> Prior to Impala 1.4.0,
+          <codeph>COMPUTE STATS</codeph> counted the number of
+          <codeph>NULL</codeph> values in each column and recorded that figure
+        in the metastore database. Because Impala does not currently use the
+          <codeph>NULL</codeph> count during query planning, Impala 1.4.0 and
+        higher speeds up the <codeph>COMPUTE STATS</codeph> statement by
+        skipping this <codeph>NULL</codeph> counting. </note>
+
+      <p id="regular_expression_whole_string">
+        The regular expression must match the entire value, not just occur somewhere inside it. Use <codeph>.*</codeph> at the beginning,
+        the end, or both if you only need to match characters anywhere in the middle. Thus, the <codeph>^</codeph> and <codeph>$</codeph>
+        atoms are often redundant, although you might already have them in your expression strings that you reuse from elsewhere.
+      </p>
+
+      <p rev="1.3.1" id="regexp_matching">
+        In Impala 1.3.1 and higher, the <codeph>REGEXP</codeph> and <codeph>RLIKE</codeph> operators now match a
+        regular expression string that occurs anywhere inside the target string, the same as if the regular
+        expression was enclosed on each side by <codeph>.*</codeph>. See
+        <xref href="../topics/impala_operators.xml#regexp"/> for examples. Previously, these operators only
+        succeeded when the regular expression matched the entire target string. This change improves compatibility
+        with the regular expression support for popular database systems. There is no change to the behavior of the
+        <codeph>regexp_extract()</codeph> and <codeph>regexp_replace()</codeph> built-in functions.
+      </p>
+
+      <p rev="1.3.1" id="insert_inherit_permissions">
+        By default, if an <codeph>INSERT</codeph> statement creates any new subdirectories underneath a partitioned
+        table, those subdirectories are assigned default HDFS permissions for the <codeph>impala</codeph> user. To
+        make each subdirectory have the same permissions as its parent directory in HDFS, specify the
+        <codeph>--insert_inherit_permissions</codeph> startup option for the <cmdname>impalad</cmdname> daemon.
+      </p>
+
+      <note id="multiple_count_distinct">
+        <p>
+          By default, Impala only allows a single <codeph>COUNT(DISTINCT <varname>columns</varname>)</codeph>
+          expression in each query.
+        </p>
+        <p>
+          If you do not need precise accuracy, you can produce an estimate of the distinct values for a column by
+          specifying <codeph>NDV(<varname>column</varname>)</codeph>; a query can contain multiple instances of
+          <codeph>NDV(<varname>column</varname>)</codeph>. To make Impala automatically rewrite
+          <codeph>COUNT(DISTINCT)</codeph> expressions to <codeph>NDV()</codeph>, enable the
+          <codeph>APPX_COUNT_DISTINCT</codeph> query option.
+        </p>
+        <p>
+          To produce the same result as multiple <codeph>COUNT(DISTINCT)</codeph> expressions, you can use the
+          following technique for queries involving a single table:
+        </p>
+<codeblock xml:space="preserve">select v1.c1 result1, v2.c1 result2 from
+  (select count(distinct col1) as c1 from t1) v1 
+    cross join
+  (select count(distinct col2) as c1 from t1) v2;
+</codeblock>
+        <p>
+          Because <codeph>CROSS JOIN</codeph> is an expensive operation, prefer to use the <codeph>NDV()</codeph>
+          technique wherever practical.
+        </p>
+      </note>
+
+      <p>
+        <ph id="union_all_vs_union">Prefer <codeph>UNION ALL</codeph> over <codeph>UNION</codeph> when you know the
+        data sets are disjoint or duplicate values are not a problem; <codeph>UNION ALL</codeph> is more efficient
+        because it avoids materializing and sorting the entire result set to eliminate duplicate values.</ph>
+      </p>
+
+      <note id="thorn">
+        The <codeph>CREATE TABLE</codeph> clauses <codeph>FIELDS TERMINATED BY</codeph>, <codeph>ESCAPED
+        BY</codeph>, and <codeph>LINES TERMINATED BY</codeph> have special rules for the string literal used for
+        their argument, because they all require a single character. You can use a regular character surrounded by
+        single or double quotation marks, an octal sequence such as <codeph>'\054'</codeph> (representing a comma),
+        or an integer in the range '-127'..'128' (with quotation marks but no backslash), which is interpreted as a
+        single-byte ASCII character. Negative values are subtracted from 256; for example, <codeph>FIELDS
+        TERMINATED BY '-2'</codeph> sets the field delimiter to ASCII code 254, the <q>Icelandic Thorn</q>
+        character used as a delimiter by some data formats.
+      </note>
+
+      <p id="sqoop_blurb">
+        <b>Sqoop considerations:</b>
+      </p>
+
+      <p id="sqoop_timestamp_caveat" rev="IMPALA-2111 CDH-37399"> If you use Sqoop to
+        convert RDBMS data to Parquet, be careful with interpreting any
+        resulting values from <codeph>DATE</codeph>, <codeph>DATETIME</codeph>,
+        or <codeph>TIMESTAMP</codeph> columns. The underlying values are
+        represented as the Parquet <codeph>INT64</codeph> type, which is
+        represented as <codeph>BIGINT</codeph> in the Impala table. The Parquet
+        values represent the time in milliseconds, while Impala interprets
+          <codeph>BIGINT</codeph> as the time in seconds. Therefore, if you have
+        a <codeph>BIGINT</codeph> column in a Parquet table that was imported
+        this way from Sqoop, divide the values by 1000 when interpreting as the
+          <codeph>TIMESTAMP</codeph> type.</p>
+
+      <p id="command_line_blurb">
+        <b>Command-line equivalent:</b>
+      </p>
+
+      <p rev="2.3.0" id="complex_types_blurb">
+        <b>Complex type considerations:</b>
+      </p>
+
+      <p id="complex_types_combo">
+        Because complex types are often used in combination,
+        for example an <codeph>ARRAY</codeph> of <codeph>STRUCT</codeph>
+        elements, if you are unfamiliar with the Impala complex types,
+        start with <xref href="../topics/impala_complex_types.xml#complex_types"/> for
+        background information and usage examples.
+      </p>
+
+      <ul id="complex_types_restrictions">
+        <li>
+          <p>
+            Columns with this da

<TRUNCATED>


[17/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_noncm_installation.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_noncm_installation.xml b/docs/topics/impala_noncm_installation.xml
new file mode 100644
index 0000000..d9c1f1a
--- /dev/null
+++ b/docs/topics/impala_noncm_installation.xml
@@ -0,0 +1,171 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="noncm_installation">
+
+  <title>Installing Impala without Cloudera Manager</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Installing"/>
+      <data name="Category" value="Administrators"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Before installing Impala manually, make sure all applicable nodes have the appropriate hardware
+      configuration, levels of operating system and CDH, and any other software prerequisites. See
+      <xref href="impala_prereqs.xml#prereqs"/> for details.
+    </p>
+
+    <p>
+      You can install Impala across many hosts or on one host:
+    </p>
+
+    <ul>
+      <li>
+        Installing Impala across multiple machines creates a distributed configuration. For best performance,
+        install Impala on <b>all</b> DataNodes.
+      </li>
+
+      <li>
+        Installing Impala on a single machine produces a pseudo-distributed cluster.
+      </li>
+    </ul>
+
+    <p>
+      <b>To install Impala on a host:</b>
+    </p>
+
+    <ol>
+      <li>
+        Install CDH as described in the Installation section of the
+<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/CDH5-Installation-Guide.html -->
+        <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/installation.html" scope="external" format="html">CDH
+        5 Installation Guide</xref>.
+      </li>
+
+      <li>
+        <p>
+          Install the Hive metastore somewhere in your cluster, as described in the Hive Installation topic in the
+<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh_ig_hive_installation.html -->
+          <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_hive_installation.html" scope="external" format="html">CDH
+          5 Installation Guide</xref>. As part of this process, you configure the Hive metastore to use an external
+          database as a metastore. Impala uses this same database for its own table metadata. You can choose either
+          a MySQL or PostgreSQL database as the metastore. The process for configuring each type of database is
+          described in the CDH Installation Guide).
+        </p>
+        <p>
+          <ph rev="upstream">Cloudera</ph> recommends setting up a Hive metastore service rather than connecting directly to the metastore
+          database; this configuration is required when running Impala under CDH 4.1. Make sure the
+          <filepath>/etc/impala/conf/hive-site.xml</filepath> file contains the following setting, substituting the
+          appropriate hostname for <varname>metastore_server_host</varname>:
+        </p>
+<codeblock>&lt;property&gt;
+&lt;name&gt;hive.metastore.uris&lt;/name&gt;
+&lt;value&gt;thrift://<varname>metastore_server_host</varname>:9083&lt;/value&gt;
+&lt;/property&gt;
+&lt;property&gt;
+&lt;name&gt;hive.metastore.client.socket.timeout&lt;/name&gt;
+&lt;value&gt;3600&lt;/value&gt;
+&lt;description&gt;MetaStore Client socket timeout in seconds&lt;/description&gt;
+&lt;/property&gt;</codeblock>
+      </li>
+
+      <li>
+        (Optional) If you installed the full Hive component on any host, you can verify that the metastore is
+        configured properly by starting the Hive console and querying for the list of available tables. Once you
+        confirm that the console starts, exit the console to continue the installation:
+<codeblock>$ hive
+Hive history file=/tmp/root/hive_job_log_root_201207272011_678722950.txt
+hive&gt; show tables;
+table1
+table2
+hive&gt; quit;
+$</codeblock>
+      </li>
+
+      <li>
+        Confirm that your package management command is aware of the Impala repository settings, as described in
+        <xref href="impala_prereqs.xml#prereqs"/>. (For CDH 4, this is a different repository than for CDH.) You
+        might need to download a repo or list file into a system directory underneath <filepath>/etc</filepath>.
+      </li>
+
+      <li>
+        Use <b>one</b> of the following sets of commands to install the Impala package:
+        <p>
+          <b>For RHEL, Oracle Linux, or CentOS systems:</b>
+        </p>
+<codeblock rev="1.2">$ sudo yum install impala             # Binaries for daemons
+$ sudo yum install impala-server      # Service start/stop script
+$ sudo yum install impala-state-store # Service start/stop script
+$ sudo yum install impala-catalog     # Service start/stop script
+</codeblock>
+        <p>
+          <b>For SUSE systems:</b>
+        </p>
+<codeblock rev="1.2">$ sudo zypper install impala             # Binaries for daemons
+$ sudo zypper install impala-server      # Service start/stop script
+$ sudo zypper install impala-state-store # Service start/stop script
+$ sudo zypper install impala-catalog     # Service start/stop script
+</codeblock>
+        <p>
+          <b>For Debian or Ubuntu systems:</b>
+        </p>
+<codeblock rev="1.2">$ sudo apt-get install impala             # Binaries for daemons
+$ sudo apt-get install impala-server      # Service start/stop script
+$ sudo apt-get install impala-state-store # Service start/stop script
+$ sudo apt-get install impala-catalog     # Service start/stop script
+</codeblock>
+        <note>
+          <ph rev="upstream">Cloudera</ph> recommends that you not install Impala on any HDFS NameNode. Installing Impala on NameNodes
+          provides no additional data locality, and executing queries with such a configuration might cause memory
+          contention and negatively impact the HDFS NameNode.
+        </note>
+      </li>
+
+      <li>
+        Copy the client <codeph>hive-site.xml</codeph>, <codeph>core-site.xml</codeph>,
+        <codeph>hdfs-site.xml</codeph>, and <codeph>hbase-site.xml</codeph> configuration files to the Impala
+        configuration directory, which defaults to <codeph>/etc/impala/conf</codeph>. Create this directory if it
+        does not already exist.
+      </li>
+
+      <li>
+        Use <b>one</b> of the following commands to install <codeph>impala-shell</codeph> on the machines from
+        which you want to issue queries. You can install <codeph>impala-shell</codeph> on any supported machine
+        that can connect to DataNodes that are running <codeph>impalad</codeph>.
+        <p>
+          <b>For RHEL/CentOS systems:</b>
+        </p>
+<codeblock>$ sudo yum install impala-shell</codeblock>
+        <p>
+          <b>For SUSE systems:</b>
+        </p>
+<codeblock>$ sudo zypper install impala-shell</codeblock>
+        <p>
+          <b>For Debian/Ubuntu systems:</b>
+        </p>
+<codeblock>$ sudo apt-get install impala-shell</codeblock>
+      </li>
+
+      <li>
+        Complete any required or recommended configuration, as described in
+        <xref href="impala_config_performance.xml#config_performance"/>. Some of these configuration changes are
+        mandatory. (They are applied automatically when you install using Cloudera Manager.)
+      </li>
+    </ol>
+
+    <p>
+      Once installation and configuration are complete, see <xref href="impala_processes.xml#processes"/> for how
+      to activate the software on the appropriate nodes in your cluster.
+    </p>
+
+    <p>
+      If this is your first time setting up and using Impala in this cluster, run through some of the exercises in
+      <xref href="impala_tutorial.xml#tutorial"/> to verify that you can do basic operations such as creating
+      tables and querying them.
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_num_nodes.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_num_nodes.xml b/docs/topics/impala_num_nodes.xml
new file mode 100644
index 0000000..8a8fe88
--- /dev/null
+++ b/docs/topics/impala_num_nodes.xml
@@ -0,0 +1,60 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="num_nodes">
+
+  <title>NUM_NODES Query Option</title>
+  <titlealts audience="PDF"><navtitle>NUM_NODES</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Troubleshooting"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">NUM_NODES query option</indexterm>
+      Limit the number of nodes that process a query, typically during debugging.
+    </p>
+
+    <p>
+      <b>Type:</b> numeric
+    </p>
+
+<p>
+      <b>Allowed values:</b> Only accepts the values 0
+      (meaning all nodes) or 1 (meaning all work is done on the coordinator node).
+</p>
+
+    <p>
+      <b>Default:</b> 0
+    </p>
+
+     <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+     <p>
+       If you are diagnosing a problem that you suspect is due to a timing issue due to
+       distributed query processing, you can set <codeph>NUM_NODES=1</codeph> to verify
+       if the problem still occurs when all the work is done on a single node.
+     </p>
+
+    <p conref="../shared/impala_common.xml#common/num_nodes_tip"/>
+
+    <note type="warning" rev="DOCS-1161">
+    <p>
+      Because this option results in increased resource utilization on a single host,
+      it could cause problems due to contention with other Impala statements or
+      high resource usage. Symptoms could include queries running slowly, exceeding the memory limit,
+      or appearing to hang. Use it only in a single-user development/test environment;
+      <b>do not</b> use it in a production environment or in a cluster with a high-concurrency
+      or high-volume or performance-critical workload.
+    </p>
+    </note>
+
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_num_scanner_threads.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_num_scanner_threads.xml b/docs/topics/impala_num_scanner_threads.xml
new file mode 100644
index 0000000..ec609d2
--- /dev/null
+++ b/docs/topics/impala_num_scanner_threads.xml
@@ -0,0 +1,35 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="num_scanner_threads">
+
+  <title>NUM_SCANNER_THREADS Query Option</title>
+  <titlealts audience="PDF"><navtitle>NUM_SCANNER_THREADS</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">NUM_SCANNER_THREADS query option</indexterm>
+      Maximum number of scanner threads (on each node) used for each query. By default, Impala uses as many cores
+      as are available (one thread per core). You might lower this value if queries are using excessive resources
+      on a busy cluster. Impala imposes a maximum value automatically, so a high value has no practical effect.
+    </p>
+
+    <p>
+      <b>Type:</b> numeric
+    </p>
+
+    <p>
+      <b>Default:</b> 0
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_odbc.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_odbc.xml b/docs/topics/impala_odbc.xml
new file mode 100644
index 0000000..c73915e
--- /dev/null
+++ b/docs/topics/impala_odbc.xml
@@ -0,0 +1,203 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="impala_odbc">
+
+  <title id="odbc">Configuring Impala to Work with ODBC</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="ODBC"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Configuring"/>
+      <data name="Category" value="Developers"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">ODBC</indexterm>
+      Third-party products can be designed to integrate with Impala using ODBC. For the best experience, ensure any
+      third-party product you intend to use is supported. Verifying support includes checking that the versions of
+      Impala, ODBC, the operating system, and the third-party product have all been approved for use together.
+      Before configuring your systems to use ODBC, download a connector.
+      You may need to sign in and accept license agreements before accessing the pages required for downloading
+      ODBC connectors.
+    </p>
+
+    <p outputclass="toc inpage"/>
+
+  </conbody>
+
+  <concept id="odbc_driver">
+    <title>Downloading the ODBC Driver</title>
+      <conbody>
+
+      <note type="important">
+        As of late 2015, most business intelligence applications are certified with the 2.x ODBC drivers.
+        Although the instructions on this page cover both the 2.x and 1.x drivers, expect to use the 2.x drivers
+        exclusively for most ODBC applications connecting to Impala.
+      </note>
+
+        <p>
+          See the
+          <xref href="http://www.cloudera.com/content/support/en/downloads/download-components/download-products.html?productID=3mDesWddUS" scope="external" format="html">downloads
+          page</xref> for a matrix of the certified driver version for different products. See the
+          <xref href="http://www.cloudera.com/content/support/en/documentation/cloudera-connectors-documentation/connector-documentation-latest.html" scope="external" format="html">documentation
+          page</xref> for installation instructions.
+        </p>
+
+      </conbody>
+    </concept>
+
+    <concept id="odbc_port">
+      <title>Configuring the ODBC Port</title>
+      <conbody>
+        <p>
+          Versions 2.5 and 2.0 of the Cloudera ODBC Connector, currently certified for some but not all BI applications,
+          use the HiveServer2 protocol, corresponding to Impala port 21050. Impala supports Kerberos authentication with
+          all the supported versions of the driver, and requires ODBC 2.05.13 for Impala or higher for LDAP username/password
+          authentication.
+        </p>
+
+        <p>
+          Version 1.x of the Cloudera ODBC Connector uses the original HiveServer1 protocol, corresponding to Impala
+          port 21000.
+        </p>
+
+      </conbody>
+    </concept>
+
+    <concept id="odbc_demo">
+      <title>Example of Setting Up an ODBC Application for Impala</title>
+      <conbody>
+
+        <p>
+          To illustrate the outline of the setup process, here is a transcript of a session to set up
+          all required drivers and a business intelligence application
+          that uses the ODBC driver, under Mac OS X. Each <codeph>.dmg</codeph> file runs a GUI-based
+          installer, first for the <xref href="http://www.iodbc.org/dataspace/iodbc/wiki/iODBC/Downloads" scope="external" format="html">underlying IODBC driver</xref> needed for non-Windows systems, then for
+          the Cloudera ODBC Connector, and finally for the BI tool itself.
+        </p>
+
+<codeblock>$ ls -1
+Cloudera-ODBC-Driver-for-Impala-Install-Guide.pdf
+BI_Tool_Installer.dmg
+iodbc-sdk-3.52.7-macosx-10.5.dmg
+ClouderaImpalaODBC.dmg
+$ open iodbc-sdk-3.52.7-macosx-10.dmg
+<varname>Install the IODBC driver using its installer</varname>
+$ open ClouderaImpalaODBC.dmg
+<varname>Install the Cloudera ODBC Connector using its installer</varname>
+$ installer_dir=$(pwd)
+$ cd /opt/cloudera/impalaodbc
+$ ls -1
+Cloudera ODBC Driver for Impala Install Guide.pdf
+Readme.txt
+Setup
+lib
+ErrorMessages
+Release Notes.txt
+Tools
+$ cd Setup
+$ ls
+odbc.ini    odbcinst.ini
+$ cp odbc.ini ~/.odbc.ini
+$ vi ~/.odbc.ini
+$ cat ~/.odbc.ini
+[ODBC]
+# Specify any global ODBC configuration here such as ODBC tracing.
+
+[ODBC Data Sources]
+Sample Cloudera Impala DSN=Cloudera ODBC Driver for Impala
+
+[Sample Cloudera Impala DSN]
+
+# Description: DSN Description.
+# This key is not necessary and is only to give a description of the data source.
+Description=Cloudera ODBC Driver for Impala DSN
+
+# Driver: The location where the ODBC driver is installed to.
+Driver=/opt/cloudera/impalaodbc/lib/universal/libclouderaimpalaodbc.dylib
+
+# The DriverUnicodeEncoding setting is only used for SimbaDM
+# When set to 1, SimbaDM runs in UTF-16 mode.
+# When set to 2, SimbaDM runs in UTF-8 mode.
+#DriverUnicodeEncoding=2
+
+# Values for HOST, PORT, KrbFQDN, and KrbServiceName should be set here.
+# They can also be specified on the connection string.
+HOST=hostname.sample.example.com
+PORT=21050
+Schema=default
+
+# The authentication mechanism.
+# 0 - No authentication (NOSASL)
+# 1 - Kerberos authentication (SASL)
+# 2 - Username authentication (SASL)
+# 3 - Username/password authentication (SASL)
+# 4 - Username/password authentication with SSL (SASL)
+# 5 - No authentication with SSL (NOSASL)
+# 6 - Username/password authentication (NOSASL)
+AuthMech=0
+
+# Kerberos related settings.
+KrbFQDN=
+KrbRealm=
+KrbServiceName=
+
+# Username/password authentication with SSL settings.
+UID=
+PWD
+CAIssuedCertNamesMismatch=1
+TrustedCerts=/opt/cloudera/impalaodbc/lib/universal/cacerts.pem
+
+# Specify the proxy user ID to use.
+#DelegationUID=
+
+# General settings
+TSaslTransportBufSize=1000
+RowsFetchedPerBlock=10000
+SocketTimeout=0
+StringColumnLength=32767
+UseNativeQuery=0
+$ pwd
+/opt/cloudera/impalaodbc/Setup
+$ cd $installer_dir
+$ open BI_Tool_Installer.dmg
+<varname>Install the BI tool using its installer</varname>
+$ ls /Applications | grep BI_Tool
+BI_Tool.app
+$ open -a BI_Tool.app
+<varname>In the BI tool, connect to a data source using port 21050</varname>
+</codeblock>
+
+    </conbody>
+  </concept>
+
+  <concept rev="2.3.0" id="jdbc_odbc_notes">
+    <title>Notes about JDBC and ODBC Interaction with Impala SQL Features</title>
+    <conbody>
+
+      <p>
+        Most Impala SQL features work equivalently through the <cmdname>impala-shell</cmdname> interpreter
+        of the JDBC or ODBC APIs. The following are some exceptions to keep in mind when switching between
+        the interactive shell and applications using the APIs:
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/proxy_jdbc_caveat"/>
+
+      <ul>
+        <li>
+          <p conref="../shared/impala_common.xml#common/jdbc_odbc_complex_types"/>
+        </li>
+        <li>
+          <p conref="../shared/impala_common.xml#common/jdbc_odbc_complex_types_views"/>
+        </li>
+      </ul>
+
+    </conbody>
+  </concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_offset.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_offset.xml b/docs/topics/impala_offset.xml
new file mode 100644
index 0000000..534f6b6
--- /dev/null
+++ b/docs/topics/impala_offset.xml
@@ -0,0 +1,66 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.2.1" id="offset">
+
+  <title>OFFSET Clause</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Reports"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      The <codeph>OFFSET</codeph> clause in a <codeph>SELECT</codeph> query causes the result set to start some
+      number of rows after the logical first item. The result set is numbered starting from zero, so <codeph>OFFSET
+      0</codeph> produces the same result as leaving out the <codeph>OFFSET</codeph> clause. Always use this clause
+      in combination with <codeph>ORDER BY</codeph> (so that it is clear which item should be first, second, and so
+      on) and <codeph>LIMIT</codeph> (so that the result set covers a bounded range, such as items 0-9, 100-199,
+      and so on).
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/limit_and_offset"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      The following example shows how you could run a <q>paging</q> query originally written for a traditional
+      database application. Because typical Impala queries process megabytes or gigabytes of data and read large
+      data files from disk each time, it is inefficient to run a separate query to retrieve each small group of
+      items. Use this technique only for compatibility while porting older applications, then rewrite the
+      application code to use a single query with a large result set, and display pages of results from the cached
+      result set.
+    </p>
+
+<codeblock>[localhost:21000] &gt; create table numbers (x int);
+[localhost:21000] &gt; insert into numbers select x from very_long_sequence;
+Inserted 1000000 rows in 1.34s
+[localhost:21000] &gt; select x from numbers order by x limit 5 offset 0;
++----+
+| x  |
++----+
+| 1  |
+| 2  |
+| 3  |
+| 4  |
+| 5  |
++----+
+[localhost:21000] &gt; select x from numbers order by x limit 5 offset 5;
++----+
+| x  |
++----+
+| 6  |
+| 7  |
+| 8  |
+| 9  |
+| 10 |
++----+
+</codeblock>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_operators.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_operators.xml b/docs/topics/impala_operators.xml
new file mode 100644
index 0000000..acd9297
--- /dev/null
+++ b/docs/topics/impala_operators.xml
@@ -0,0 +1,1592 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="operators">
+
+  <title>SQL Operators</title>
+
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">operators</indexterm>
+      SQL operators are a class of comparison functions that are widely used within the <codeph>WHERE</codeph> clauses of
+      <codeph>SELECT</codeph> statements.
+    </p>
+
+    <p outputclass="toc inpage"/>
+
+  </conbody>
+
+  <concept rev="1.4.0" id="arithmetic_operators">
+
+    <title>Arithmetic Operators</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">arithmetic operators</indexterm>
+        The arithmetic operators use expressions with a left-hand argument, the operator, and then (in most cases) a right-hand argument.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock><varname>left_hand_arg</varname> <varname>binary_operator</varname> <varname>right_hand_arg</varname>
+<varname>unary_operator</varname> <varname>single_arg</varname>
+</codeblock>
+
+      <ul>
+        <li>
+          <codeph>+</codeph> and <codeph>-</codeph>: Can be used either as unary or binary operators.
+          <ul>
+            <li>
+              <p>
+                With unary notation, such as <codeph>+5</codeph>, <codeph>-2.5</codeph>, or <codeph>-<varname>col_name</varname></codeph>,
+                they multiply their single numeric argument by <codeph>+1</codeph> or <codeph>-1</codeph>. Therefore, unary
+                <codeph>+</codeph> returns its argument unchanged, while unary <codeph>-</codeph> flips the sign of its argument. Although
+                you can double up these operators in expressions such as <codeph>++5</codeph> (always positive) or <codeph>-+2</codeph> or
+                <codeph>+-2</codeph> (both always negative), you cannot double the unary minus operator because <codeph>--</codeph> is
+                interpreted as the start of a comment. (You can use a double unary minus operator if you separate the <codeph>-</codeph>
+                characters, for example with a space or parentheses.)
+              </p>
+            </li>
+
+            <li>
+              <p>
+                With binary notation, such as <codeph>2+2</codeph>, <codeph>5-2.5</codeph>, or <codeph><varname>col1</varname> +
+                <varname>col2</varname></codeph>, they add or subtract respectively the right-hand argument to (or from) the left-hand
+                argument. Both arguments must be of numeric types.
+              </p>
+            </li>
+          </ul>
+        </li>
+
+        <li>
+          <p>
+            <codeph>*</codeph> and <codeph>/</codeph>: Multiplication and division respectively. Both arguments must be of numeric types.
+          </p>
+
+          <p>
+            When multiplying, the shorter argument is promoted if necessary (such as <codeph>SMALLINT</codeph> to <codeph>INT</codeph> or
+            <codeph>BIGINT</codeph>, or <codeph>FLOAT</codeph> to <codeph>DOUBLE</codeph>), and then the result is promoted again to the
+            next larger type. Thus, multiplying a <codeph>TINYINT</codeph> and an <codeph>INT</codeph> produces a <codeph>BIGINT</codeph>
+            result. Multiplying a <codeph>FLOAT</codeph> and a <codeph>FLOAT</codeph> produces a <codeph>DOUBLE</codeph> result. Multiplying
+            a <codeph>FLOAT</codeph> and a <codeph>DOUBLE</codeph> or a <codeph>DOUBLE</codeph> and a <codeph>DOUBLE</codeph> produces a
+            <codeph>DECIMAL(38,17)</codeph>, because <codeph>DECIMAL</codeph> values can represent much larger and more precise values than
+            <codeph>DOUBLE</codeph>.
+          </p>
+
+          <p>
+            When dividing, Impala always treats the arguments and result as <codeph>DOUBLE</codeph> values to avoid losing precision. If you
+            need to insert the results of a division operation into a <codeph>FLOAT</codeph> column, use the <codeph>CAST()</codeph>
+            function to convert the result to the correct type.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            <codeph>%</codeph>: Modulo operator. Returns the remainder of the left-hand argument divided by the right-hand argument. Both
+            arguments must be of one of the integer types.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            <codeph>&amp;</codeph>, <codeph>|</codeph>, <codeph>~</codeph>, and <codeph>^</codeph>: Bitwise operators that return the
+            logical AND, logical OR, <codeph>NOT</codeph>, or logical XOR (exclusive OR) of their argument values. Both arguments must be of
+            one of the integer types. If the arguments are of different type, the argument with the smaller type is implicitly extended to
+            match the argument with the longer type.
+          </p>
+        </li>
+      </ul>
+
+      <p>
+        You can chain a sequence of arithmetic expressions, optionally grouping them with parentheses.
+      </p>
+
+      <p>
+        The arithmetic operators generally do not have equivalent calling conventions using functional notation. For example, prior to
+        Impala 2.2.0 / CDH 5.4.0, there is no <codeph>MOD()</codeph> function equivalent to the <codeph>%</codeph> modulo operator.
+        Conversely, there are some arithmetic functions that do not have a corresponding operator. For example, for exponentiation you use
+        the <codeph>POW()</codeph> function, but there is no <codeph>**</codeph> exponentiation operator. See
+        <xref href="impala_math_functions.xml#math_functions"/> for the arithmetic functions you can use.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_aggregation_explanation"/>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_aggregation_example"/>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_caveat_no_operator"/>
+
+      <p rev="2.3.0">
+        The following example shows how to do an arithmetic operation using a numeric field of a <codeph>STRUCT</codeph> type that is an
+        item within an <codeph>ARRAY</codeph> column. Once the scalar numeric value <codeph>R_NATIONKEY</codeph> is extracted, it can be
+        used in an arithmetic expression, such as multiplying by 10:
+      </p>
+
+<codeblock rev="2.3.0">
+-- The SMALLINT is a field within an array of structs.
+describe region;
++-------------+-------------------------+---------+
+| name        | type                    | comment |
++-------------+-------------------------+---------+
+| r_regionkey | smallint                |         |
+| r_name      | string                  |         |
+| r_comment   | string                  |         |
+| r_nations   | array&lt;struct&lt;           |         |
+|             |   n_nationkey:smallint, |         |
+|             |   n_name:string,        |         |
+|             |   n_comment:string      |         |
+|             | &gt;&gt;                      |         |
++-------------+-------------------------+---------+
+
+-- When we refer to the scalar value using dot notation,
+-- we can use arithmetic and comparison operators on it
+-- like any other number.
+select r_name, nation.item.n_name, nation.item.n_nationkey * 10
+  from region, region.r_nations as nation
+where nation.item.n_nationkey &lt; 5;
++-------------+-------------+------------------------------+
+| r_name      | item.n_name | nation.item.n_nationkey * 10 |
++-------------+-------------+------------------------------+
+| AMERICA     | CANADA      | 30                           |
+| AMERICA     | BRAZIL      | 20                           |
+| AMERICA     | ARGENTINA   | 10                           |
+| MIDDLE EAST | EGYPT       | 40                           |
+| AFRICA      | ALGERIA     | 0                            |
++-------------+-------------+------------------------------+
+</codeblock>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="between">
+
+    <title>BETWEEN Operator</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">BETWEEN operator</indexterm>
+        In a <codeph>WHERE</codeph> clause, compares an expression to both a lower and upper bound. The comparison is successful is the
+        expression is greater than or equal to the lower bound, and less than or equal to the upper bound. If the bound values are switched,
+        so the lower bound is greater than the upper bound, does not match any values.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock><varname>expression</varname> BETWEEN <varname>lower_bound</varname> AND <varname>upper_bound</varname></codeblock>
+
+      <p>
+        <b>Data types:</b> Typically used with numeric data types. Works with any data type, although not very practical for
+        <codeph>BOOLEAN</codeph> values. (<codeph>BETWEEN false AND true</codeph> will match all <codeph>BOOLEAN</codeph> values.) Use
+        <codeph>CAST()</codeph> if necessary to ensure the lower and upper bound values are compatible types. Call string or date/time
+        functions if necessary to extract or transform the relevant portion to compare, especially if the value can be transformed into a
+        number.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p>
+        Be careful when using short string operands. A longer string that starts with the upper bound value will not be included, because it
+        is considered greater than the upper bound. For example, <codeph>BETWEEN 'A' and 'M'</codeph> would not match the string value
+        <codeph>'Midway'</codeph>. Use functions such as <codeph>upper()</codeph>, <codeph>lower()</codeph>, <codeph>substr()</codeph>,
+        <codeph>trim()</codeph>, and so on if necessary to ensure the comparison works as expected.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_caveat_no_operator"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>-- Retrieve data for January through June, inclusive.
+select c1 from t1 where month <b>between 1 and 6</b>;
+
+-- Retrieve data for names beginning with 'A' through 'M' inclusive.
+-- Only test the first letter to ensure all the values starting with 'M' are matched.
+-- Do a case-insensitive comparison to match names with various capitalization conventions.
+select last_name from customers where upper(substr(last_name,1,1)) <b>between 'A' and 'M'</b>;
+
+-- Retrieve data for only the first week of each month.
+select count(distinct visitor_id)) from web_traffic where dayofmonth(when_viewed) <b>between 1 and 7</b>;</codeblock>
+
+      <p rev="2.3.0">
+        The following example shows how to do a <codeph>BETWEEN</codeph> comparison using a numeric field of a <codeph>STRUCT</codeph> type
+        that is an item within an <codeph>ARRAY</codeph> column. Once the scalar numeric value <codeph>R_NATIONKEY</codeph> is extracted, it
+        can be used in a comparison operator:
+      </p>
+
+<codeblock rev="2.3.0">
+-- The SMALLINT is a field within an array of structs.
+describe region;
++-------------+-------------------------+---------+
+| name        | type                    | comment |
++-------------+-------------------------+---------+
+| r_regionkey | smallint                |         |
+| r_name      | string                  |         |
+| r_comment   | string                  |         |
+| r_nations   | array&lt;struct&lt;           |         |
+|             |   n_nationkey:smallint, |         |
+|             |   n_name:string,        |         |
+|             |   n_comment:string      |         |
+|             | &gt;&gt;                      |         |
++-------------+-------------------------+---------+
+
+-- When we refer to the scalar value using dot notation,
+-- we can use arithmetic and comparison operators on it
+-- like any other number.
+select r_name, nation.item.n_name, nation.item.n_nationkey
+from region, region.r_nations as nation
+where nation.item.n_nationkey between 3 and 5
++-------------+-------------+------------------+
+| r_name      | item.n_name | item.n_nationkey |
++-------------+-------------+------------------+
+| AMERICA     | CANADA      | 3                |
+| MIDDLE EAST | EGYPT       | 4                |
+| AFRICA      | ETHIOPIA    | 5                |
++-------------+-------------+------------------+
+</codeblock>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="comparison_operators">
+
+    <title>Comparison Operators</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">comparison operators</indexterm>
+        Impala supports the familiar comparison operators for checking equality and sort order for the column data types:
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock><varname>left_hand_expression</varname> <varname>comparison_operator</varname> <varname>right_hand_expression</varname></codeblock>
+
+      <ul>
+        <li>
+          <codeph>=</codeph>, <codeph>!=</codeph>, <codeph>&lt;&gt;</codeph>: apply to all types.
+        </li>
+
+        <li>
+          <codeph>&lt;</codeph>, <codeph>&lt;=</codeph>, <codeph>&gt;</codeph>, <codeph>&gt;=</codeph>: apply to all types; for
+          <codeph>BOOLEAN</codeph>, <codeph>TRUE</codeph> is considered greater than <codeph>FALSE</codeph>.
+        </li>
+      </ul>
+
+      <p>
+        <b>Alternatives:</b>
+      </p>
+
+      <p>
+        The <codeph>IN</codeph> and <codeph>BETWEEN</codeph> operators provide shorthand notation for expressing combinations of equality,
+        less than, and greater than comparisons with a single operator.
+      </p>
+
+      <p>
+        Because comparing any value to <codeph>NULL</codeph> produces <codeph>NULL</codeph> rather than <codeph>TRUE</codeph> or
+        <codeph>FALSE</codeph>, use the <codeph>IS NULL</codeph> and <codeph>IS NOT NULL</codeph> operators to check if a value is
+        <codeph>NULL</codeph> or not.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_caveat_no_operator"/>
+
+      <p rev="2.3.0">
+        The following example shows how to do an arithmetic operation using a numeric field of a <codeph>STRUCT</codeph> type that is an
+        item within an <codeph>ARRAY</codeph> column. Once the scalar numeric value <codeph>R_NATIONKEY</codeph> is extracted, it can be
+        used with a comparison operator such as <codeph>&lt;</codeph>:
+      </p>
+
+<codeblock rev="2.3.0">
+-- The SMALLINT is a field within an array of structs.
+describe region;
++-------------+-------------------------+---------+
+| name        | type                    | comment |
++-------------+-------------------------+---------+
+| r_regionkey | smallint                |         |
+| r_name      | string                  |         |
+| r_comment   | string                  |         |
+| r_nations   | array&lt;struct&lt;           |         |
+|             |   n_nationkey:smallint, |         |
+|             |   n_name:string,        |         |
+|             |   n_comment:string      |         |
+|             | &gt;&gt;                      |         |
++-------------+-------------------------+---------+
+
+-- When we refer to the scalar value using dot notation,
+-- we can use arithmetic and comparison operators on it
+-- like any other number.
+select r_name, nation.item.n_name, nation.item.n_nationkey
+from region, region.r_nations as nation
+where nation.item.n_nationkey &lt; 5
++-------------+-------------+------------------+
+| r_name      | item.n_name | item.n_nationkey |
++-------------+-------------+------------------+
+| AMERICA     | CANADA      | 3                |
+| AMERICA     | BRAZIL      | 2                |
+| AMERICA     | ARGENTINA   | 1                |
+| MIDDLE EAST | EGYPT       | 4                |
+| AFRICA      | ALGERIA     | 0                |
++-------------+-------------+------------------+
+</codeblock>
+
+    </conbody>
+
+  </concept>
+
+  <concept audience="Cloudera" rev="2.1.0" id="except">
+
+    <title>EXCEPT Operator</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">EXCEPT operator</indexterm>
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.0.0" id="exists">
+
+    <title>EXISTS Operator</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">EXISTS operator</indexterm>
+
+        <indexterm audience="Cloudera">NOT EXISTS operator</indexterm>
+        The <codeph>EXISTS</codeph> operator tests whether a subquery returns any results. You typically use it to find values from one
+        table that have corresponding values in another table.
+      </p>
+
+      <p>
+        The converse, <codeph>NOT EXISTS</codeph>, helps to find all the values from one table that do not have any corresponding values in
+        another table.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>EXISTS (<varname>subquery</varname>)
+NOT EXISTS (<varname>subquery</varname>)
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p>
+        The subquery can refer to a different table than the outer query block, or the same table. For example, you might use
+        <codeph>EXISTS</codeph> or <codeph>NOT EXISTS</codeph> to check the existence of parent/child relationships between two columns of
+        the same table.
+      </p>
+
+      <p>
+        You can also use operators and function calls within the subquery to test for other kinds of relationships other than strict
+        equality. For example, you might use a call to <codeph>COUNT()</codeph> in the subquery to check whether the number of matching
+        values is higher or lower than some limit. You might call a UDF in the subquery to check whether values in one table matches a
+        hashed representation of those same values in a different table.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/null_blurb"/>
+
+      <p>
+        If the subquery returns any value at all (even <codeph>NULL</codeph>), <codeph>EXISTS</codeph> returns <codeph>TRUE</codeph> and
+        <codeph>NOT EXISTS</codeph> returns false.
+      </p>
+
+      <p>
+        The following example shows how even when the subquery returns only <codeph>NULL</codeph> values, <codeph>EXISTS</codeph> still
+        returns <codeph>TRUE</codeph> and thus matches all the rows from the table in the outer query block.
+      </p>
+
+<codeblock>[localhost:21000] &gt; create table all_nulls (x int);
+[localhost:21000] &gt; insert into all_nulls values (null), (null), (null);
+[localhost:21000] &gt; select y from t2 where exists (select x from all_nulls);
++---+
+| y |
++---+
+| 2 |
+| 4 |
+| 6 |
++---+
+</codeblock>
+
+      <p>
+        However, if the table in the subquery is empty and so the subquery returns an empty result set, <codeph>EXISTS</codeph> returns
+        <codeph>FALSE</codeph>:
+      </p>
+
+<codeblock>[localhost:21000] &gt; create table empty (x int);
+[localhost:21000] &gt; select y from t2 where exists (select x from empty);
+[localhost:21000] &gt;
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/added_in_20"/>
+
+      <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/subquery_no_limit"/>
+
+      <p rev="IMPALA-3232">
+        Prior to <keyword keyref="impala26_full"/>,
+        the <codeph>NOT EXISTS</codeph> operator required a correlated subquery.
+        In <keyword keyref="impala26_full"/> and higher, <codeph>NOT EXISTS</codeph> works with
+        uncorrelated queries also.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_caveat_no_operator"/>
+
+<!-- To do: construct an EXISTS / NOT EXISTS example for complex types. -->
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+<!-- Maybe turn this into a conref if the same set of tables gets used for subqueries, EXISTS, other places. -->
+<!-- Yes, the material was reused under Subqueries for anti-joins. -->
+        The following examples refer to these simple tables containing small sets of integers or strings:
+<codeblock>[localhost:21000] &gt; create table t1 (x int);
+[localhost:21000] &gt; insert into t1 values (1), (2), (3), (4), (5), (6);
+
+[localhost:21000] &gt; create table t2 (y int);
+[localhost:21000] &gt; insert into t2 values (2), (4), (6);
+
+[localhost:21000] &gt; create table t3 (z int);
+[localhost:21000] &gt; insert into t3 values (1), (3), (5);
+
+[localhost:21000] &gt; create table month_names (m string);
+[localhost:21000] &gt; insert into month_names values
+                  &gt; ('January'), ('February'), ('March'),
+                  &gt; ('April'), ('May'), ('June'), ('July'),
+                  &gt; ('August'), ('September'), ('October'),
+                  &gt; ('November'), ('December');
+</codeblock>
+      </p>
+
+      <p>
+        The following example shows a correlated subquery that finds all the values in one table that exist in another table. For each value
+        <codeph>X</codeph> from <codeph>T1</codeph>, the query checks if the <codeph>Y</codeph> column of <codeph>T2</codeph> contains an
+        identical value, and the <codeph>EXISTS</codeph> operator returns <codeph>TRUE</codeph> or <codeph>FALSE</codeph> as appropriate in
+        each case.
+      </p>
+
+<codeblock>localhost:21000] &gt; select x from t1 where exists (select y from t2 where t1.x = y);
++---+
+| x |
++---+
+| 2 |
+| 4 |
+| 6 |
++---+
+</codeblock>
+
+      <p>
+        An uncorrelated query is less interesting in this case. Because the subquery always returns <codeph>TRUE</codeph>, all rows from
+        <codeph>T1</codeph> are returned. If the table contents where changed so that the subquery did not match any rows, none of the rows
+        from <codeph>T1</codeph> would be returned.
+      </p>
+
+<codeblock>[localhost:21000] &gt; select x from t1 where exists (select y from t2 where y &gt; 5);
++---+
+| x |
++---+
+| 1 |
+| 2 |
+| 3 |
+| 4 |
+| 5 |
+| 6 |
++---+
+</codeblock>
+
+      <p>
+        The following example shows how an uncorrelated subquery can test for the existence of some condition within a table. By using
+        <codeph>LIMIT 1</codeph> or an aggregate function, the query returns a single result or no result based on whether the subquery
+        matches any rows. Here, we know that <codeph>T1</codeph> and <codeph>T2</codeph> contain some even numbers, but <codeph>T3</codeph>
+        does not.
+      </p>
+
+<codeblock>[localhost:21000] &gt; select "contains an even number" from t1 where exists (select x from t1 where x % 2 = 0) limit 1;
++---------------------------+
+| 'contains an even number' |
++---------------------------+
+| contains an even number   |
++---------------------------+
+[localhost:21000] &gt; select "contains an even number" as assertion from t1 where exists (select x from t1 where x % 2 = 0) limit 1;
++-------------------------+
+| assertion               |
++-------------------------+
+| contains an even number |
++-------------------------+
+[localhost:21000] &gt; select "contains an even number" as assertion from t2 where exists (select x from t2 where y % 2 = 0) limit 1;
+ERROR: AnalysisException: couldn't resolve column reference: 'x'
+[localhost:21000] &gt; select "contains an even number" as assertion from t2 where exists (select y from t2 where y % 2 = 0) limit 1;
++-------------------------+
+| assertion               |
++-------------------------+
+| contains an even number |
++-------------------------+
+[localhost:21000] &gt; select "contains an even number" as assertion from t3 where exists (select z from t3 where z % 2 = 0) limit 1;
+[localhost:21000] &gt;
+</codeblock>
+
+      <p>
+        The following example finds numbers in one table that are 1 greater than numbers from another table. The <codeph>EXISTS</codeph>
+        notation is simpler than an equivalent <codeph>CROSS JOIN</codeph> between the tables. (The example then also illustrates how the
+        same test could be performed using an <codeph>IN</codeph> operator.)
+      </p>
+
+<codeblock>[localhost:21000] &gt; select x from t1 where exists (select y from t2 where x = y + 1);
++---+
+| x |
++---+
+| 3 |
+| 5 |
++---+
+[localhost:21000] &gt; select x from t1 where x in (select y + 1 from t2);
++---+
+| x |
++---+
+| 3 |
+| 5 |
++---+
+</codeblock>
+
+      <p>
+        The following example finds values from one table that do not exist in another table.
+      </p>
+
+<codeblock>[localhost:21000] &gt; select x from t1 where not exists (select y from t2 where x = y);
++---+
+| x |
++---+
+| 1 |
+| 3 |
+| 5 |
++---+
+</codeblock>
+
+      <p>
+        The following example uses the <codeph>NOT EXISTS</codeph> operator to find all the leaf nodes in tree-structured data. This
+        simplified <q>tree of life</q> has multiple levels (class, order, family, and so on), with each item pointing upward through a
+        <codeph>PARENT</codeph> pointer. The example runs an outer query and a subquery on the same table, returning only those items whose
+        <codeph>ID</codeph> value is <i>not</i> referenced by the <codeph>PARENT</codeph> of any other item.
+      </p>
+
+<codeblock>[localhost:21000] &gt; create table tree (id int, parent int, name string);
+[localhost:21000] &gt; insert overwrite tree values
+                  &gt; (0, null, "animals"),
+                  &gt; (1, 0, "placentals"),
+                  &gt; (2, 0, "marsupials"),
+                  &gt; (3, 1, "bats"),
+                  &gt; (4, 1, "cats"),
+                  &gt; (5, 2, "kangaroos"),
+                  &gt; (6, 4, "lions"),
+                  &gt; (7, 4, "tigers"),
+                  &gt; (8, 5, "red kangaroo"),
+                  &gt; (9, 2, "wallabies");
+[localhost:21000] &gt; select name as "leaf node" from tree one
+                  &gt; where not exists (select parent from tree two where one.id = two.parent);
++--------------+
+| leaf node    |
++--------------+
+| bats         |
+| lions        |
+| tigers       |
+| red kangaroo |
+| wallabies    |
++--------------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_subqueries.xml#subqueries"/>
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.4.0" id="ilike">
+
+    <title>ILIKE Operator</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">ILIKE operator</indexterm>
+        A case-insensitive comparison operator for <codeph>STRING</codeph> data, with basic wildcard capability using <codeph>_</codeph> to match a single
+        character and <codeph>%</codeph> to match multiple characters. The argument expression must match the entire string value.
+        Typically, it is more efficient to put any <codeph>%</codeph> wildcard match at the end of the string.
+      </p>
+
+      <p>
+        This operator, available in <keyword keyref="impala25_full"/> and higher, is the equivalent of the <codeph>LIKE</codeph> operator,
+        but with case-insensitive comparisons.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock><varname>string_expression</varname> ILIKE <varname>wildcard_expression</varname>
+<varname>string_expression</varname> NOT ILIKE <varname>wildcard_expression</varname>
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_caveat_no_operator"/>
+
+<!-- To do: construct a LIKE example for complex types. -->
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+      <p>
+        In the following examples, strings that are the same except for differences in uppercase
+        and lowercase match successfully with <codeph>ILIKE</codeph>, but do not match
+        with <codeph>LIKE</codeph>:
+      </p>
+
+<codeblock>select 'fooBar' ilike 'FOOBAR';
++-------------------------+
+| 'foobar' ilike 'foobar' |
++-------------------------+
+| true                    |
++-------------------------+
+
+select 'fooBar' like 'FOOBAR';
++------------------------+
+| 'foobar' like 'foobar' |
++------------------------+
+| false                  |
++------------------------+
+
+select 'FOOBAR' ilike 'f%';
++---------------------+
+| 'foobar' ilike 'f%' |
++---------------------+
+| true                |
++---------------------+
+
+select 'FOOBAR' like 'f%';
++--------------------+
+| 'foobar' like 'f%' |
++--------------------+
+| false              |
++--------------------+
+
+select 'ABCXYZ' not ilike 'ab_xyz';
++-----------------------------+
+| not 'abcxyz' ilike 'ab_xyz' |
++-----------------------------+
+| false                       |
++-----------------------------+
+
+select 'ABCXYZ' not like 'ab_xyz';
++----------------------------+
+| not 'abcxyz' like 'ab_xyz' |
++----------------------------+
+| true                       |
++----------------------------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p rev="2.5.0">
+        For case-sensitive comparisons, see <xref href="impala_operators.xml#like"/>.
+        For a more general kind of search operator using regular expressions, see <xref href="impala_operators.xml#regexp"/>
+        or its case-insensitive counterpart <xref href="impala_operators.xml#iregexp"/>.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="in">
+
+    <title>IN Operator</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">IN operator</indexterm>
+
+        <indexterm audience="Cloudera">NOT IN operator</indexterm>
+        The <codeph>IN</codeph> operator compares an argument value to a set of values, and returns <codeph>TRUE</codeph> if the argument
+        matches any value in the set. The <codeph>NOT IN</codeph> operator reverses the comparison, and checks if the argument value is not
+        part of a set of values.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock rev="2.0.0"><varname>expression</varname> IN (<varname>expression</varname> [, <varname>expression</varname>])
+<varname>expression</varname> IN (<varname>subquery</varname>)
+
+<varname>expression</varname> NOT IN (<varname>expression</varname> [, <varname>expression</varname>])
+<varname>expression</varname> NOT IN (<varname>subquery</varname>)
+</codeblock>
+
+      <p>
+        The left-hand expression and the set of comparison values must be of compatible types.
+      </p>
+
+      <p>
+        The left-hand expression must consist only of a single value, not a tuple. Although the left-hand expression is typically a column
+        name, it could also be some other value. For example, the <codeph>WHERE</codeph> clauses <codeph>WHERE id IN (5)</codeph> and
+        <codeph>WHERE 5 IN (id)</codeph> produce the same results.
+      </p>
+
+      <p rev="CDH-40407">
+        The set of values to check against can be specified as constants, function calls, column names, or other expressions in the query
+        text. The maximum number of expressions in the <codeph>IN</codeph> list is 9999. (The maximum number of elements of
+        a single expression is 10,000 items, and the <codeph>IN</codeph> operator itself counts as one.)
+      </p>
+
+      <p rev="2.0.0">
+        In Impala 2.0 and higher, the set of values can also be generated by a subquery. <codeph>IN</codeph> can evaluate an unlimited
+        number of results using a subquery.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p>
+        Any expression using the <codeph>IN</codeph> operator could be rewritten as a series of equality tests connected with
+        <codeph>OR</codeph>, but the <codeph>IN</codeph> syntax is often clearer, more concise, and easier for Impala to optimize. For
+        example, with partitioned tables, queries frequently use <codeph>IN</codeph> clauses to filter data by comparing the partition key
+        columns to specific values.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/null_blurb"/>
+
+      <p>
+        If there really is a matching non-null value, <codeph>IN</codeph> returns <codeph>TRUE</codeph>:
+      </p>
+
+<codeblock>[localhost:21000] &gt; select 1 in (1,null,2,3);
++----------------------+
+| 1 in (1, null, 2, 3) |
++----------------------+
+| true                 |
++----------------------+
+[localhost:21000] &gt; select 1 not in (1,null,2,3);
++--------------------------+
+| 1 not in (1, null, 2, 3) |
++--------------------------+
+| false                    |
++--------------------------+
+</codeblock>
+
+      <p>
+        If the searched value is not found in the comparison values, and the comparison values include <codeph>NULL</codeph>, the result is
+        <codeph>NULL</codeph>:
+      </p>
+
+<codeblock>[localhost:21000] &gt; select 5 in (1,null,2,3);
++----------------------+
+| 5 in (1, null, 2, 3) |
++----------------------+
+| NULL                 |
++----------------------+
+[localhost:21000] &gt; select 5 not in (1,null,2,3);
++--------------------------+
+| 5 not in (1, null, 2, 3) |
++--------------------------+
+| NULL                     |
++--------------------------+
+[localhost:21000] &gt; select 1 in (null);
++-------------+
+| 1 in (null) |
++-------------+
+| NULL        |
++-------------+
+[localhost:21000] &gt; select 1 not in (null);
++-----------------+
+| 1 not in (null) |
++-----------------+
+| NULL            |
++-----------------+
+</codeblock>
+
+      <p>
+        If the left-hand argument is <codeph>NULL</codeph>, <codeph>IN</codeph> always returns <codeph>NULL</codeph>. This rule applies even
+        if the comparison values include <codeph>NULL</codeph>.
+      </p>
+
+<codeblock>[localhost:21000] &gt; select null in (1,2,3);
++-------------------+
+| null in (1, 2, 3) |
++-------------------+
+| NULL              |
++-------------------+
+[localhost:21000] &gt; select null not in (1,2,3);
++-----------------------+
+| null not in (1, 2, 3) |
++-----------------------+
+| NULL                  |
++-----------------------+
+[localhost:21000] &gt; select null in (null);
++----------------+
+| null in (null) |
++----------------+
+| NULL           |
++----------------+
+[localhost:21000] &gt; select null not in (null);
++--------------------+
+| null not in (null) |
++--------------------+
+| NULL               |
++--------------------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/enhanced_in_20"/>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_caveat_no_operator"/>
+
+      <p rev="2.3.0">
+        The following example shows how to do an arithmetic operation using a numeric field of a <codeph>STRUCT</codeph> type that is an
+        item within an <codeph>ARRAY</codeph> column. Once the scalar numeric value <codeph>R_NATIONKEY</codeph> is extracted, it can be
+        used in an arithmetic expression, such as multiplying by 10:
+      </p>
+
+<codeblock rev="2.3.0">
+-- The SMALLINT is a field within an array of structs.
+describe region;
++-------------+-------------------------+---------+
+| name        | type                    | comment |
++-------------+-------------------------+---------+
+| r_regionkey | smallint                |         |
+| r_name      | string                  |         |
+| r_comment   | string                  |         |
+| r_nations   | array&lt;struct&lt;           |         |
+|             |   n_nationkey:smallint, |         |
+|             |   n_name:string,        |         |
+|             |   n_comment:string      |         |
+|             | &gt;&gt;                      |         |
++-------------+-------------------------+---------+
+
+-- When we refer to the scalar value using dot notation,
+-- we can use arithmetic and comparison operators on it
+-- like any other number.
+select r_name, nation.item.n_name, nation.item.n_nationkey
+from region, region.r_nations as nation
+where nation.item.n_nationkey in (1,3,5)
++---------+-------------+------------------+
+| r_name  | item.n_name | item.n_nationkey |
++---------+-------------+------------------+
+| AMERICA | CANADA      | 3                |
+| AMERICA | ARGENTINA   | 1                |
+| AFRICA  | ETHIOPIA    | 5                |
++---------+-------------+------------------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/subquery_no_limit"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>-- Using IN is concise and self-documenting.
+SELECT * FROM t1 WHERE c1 IN (1,2,10);
+-- Equivalent to series of = comparisons ORed together.
+SELECT * FROM t1 WHERE c1 = 1 OR c1 = 2 OR c1 = 10;
+
+SELECT c1 AS "starts with vowel" FROM t2 WHERE upper(substr(c1,1,1)) IN ('A','E','I','O','U');
+
+SELECT COUNT(DISTINCT(visitor_id)) FROM web_traffic WHERE month IN ('January','June','July');</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_subqueries.xml#subqueries"/>
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept audience="Cloudera" rev="2.1.0" id="intersect">
+
+    <title>INTERSECT Operator</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">INTERSECT operator</indexterm>
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.5.0" id="iregexp">
+
+    <title>IREGEXP Operator</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">IREGEXP operator</indexterm>
+        Tests whether a value matches a regular expression, using case-insensitive string comparisons.
+        Uses the POSIX regular expression syntax where <codeph>^</codeph> and
+        <codeph>$</codeph> match the beginning and end of the string, <codeph>.</codeph> represents any single character, <codeph>*</codeph>
+        represents a sequence of zero or more items, <codeph>+</codeph> represents a sequence of one or more items, <codeph>?</codeph>
+        produces a non-greedy match, and so on.
+      </p>
+
+      <p>
+        This operator, available in <keyword keyref="impala25_full"/> and higher, is the equivalent of the <codeph>REGEXP</codeph> operator,
+        but with case-insensitive comparisons.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock><varname>string_expression</varname> IREGEXP <varname>regular_expression</varname>
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/regular_expression_whole_string"/>
+
+<!-- Currently, there isn't any IRLIKE synonym, so REGEXP and IREGEXP are different in that respect.
+     I pinged IMPALA-1787 to check if that's intentional.
+      <p>
+        The <codeph>IRLIKE</codeph> operator is a synonym for <codeph>IREGEXP</codeph>.
+      </p>
+-->
+
+      <p rev="2.5.0">
+        The <codeph>|</codeph> symbol is the alternation operator, typically used within <codeph>()</codeph> to match different sequences.
+        The <codeph>()</codeph> groups do not allow backreferences. To retrieve the part of a value matched within a <codeph>()</codeph>
+        section, use the <codeph><xref href="impala_string_functions.xml#string_functions/regexp_extract">regexp_extract()</xref></codeph>
+        built-in function. (Currently, there is not any case-insensitive equivalent for the <codeph>regexp_extract()</codeph> function.)
+      </p>
+
+      <note rev="1.3.1">
+        <p rev="1.3.1" conref="../shared/impala_common.xml#common/regexp_matching"/>
+      </note>
+
+      <p conref="../shared/impala_common.xml#common/regexp_re2"/>
+
+      <p conref="../shared/impala_common.xml#common/regexp_re2_warning"/>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_caveat_no_operator"/>
+
+<!-- To do: construct a REGEXP example for complex types. -->
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following examples demonstrate the syntax for the <codeph>IREGEXP</codeph> operator.
+      </p>
+
+<codeblock>select 'abcABCaabbcc' iregexp '^[a-c]+$';
++---------------------------------+
+| 'abcabcaabbcc' iregexp '[a-c]+' |
++---------------------------------+
+| true                            |
++---------------------------------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_operators.xml#regexp"/>
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.5.0 IMPALA-2147" id="is_distinct_from">
+
+    <title id="is_distinct">IS DISTINCT FROM Operator</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">IS DISTINCT FROM operator</indexterm>
+
+        <indexterm audience="Cloudera">IS NOT DISTINCT FROM operator</indexterm>
+        The <codeph>IS DISTINCT FROM</codeph> operator, and its converse the <codeph>IS NOT DISTINCT FROM</codeph> operator, test whether or
+        not values are identical. <codeph>IS NOT DISTINCT FROM</codeph> is similar to the <codeph>=</codeph> operator, and <codeph>IS
+        DISTINCT FROM</codeph> is similar to the <codeph>!=</codeph> operator, except that <codeph>NULL</codeph> values are treated as
+        identical. Therefore, <codeph>IS NOT DISTINCT FROM</codeph> returns <codeph>true</codeph> rather than <codeph>NULL</codeph>, and
+        <codeph>IS DISTINCT FROM</codeph> returns <codeph>false</codeph> rather than <codeph>NULL</codeph>, when comparing two
+        <codeph>NULL</codeph> values. If one of the values being compared is <codeph>NULL</codeph> and the other is not, <codeph>IS DISTINCT
+        FROM</codeph> returns <codeph>true</codeph> and <codeph>IS NOT DISTINCT FROM</codeph> returns <codeph>false</codeph>, again instead
+        of returning <codeph>NULL</codeph> in both cases.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock><varname>expression1</varname> IS DISTINCT FROM <varname>expression2</varname>
+
+<varname>expression1</varname> IS NOT DISTINCT FROM <varname>expression2</varname>
+<varname>expression1</varname> &lt;=&gt; <varname>expression2</varname>
+</codeblock>
+
+      <p>
+        The operator <codeph>&lt;=&gt;</codeph> is an alias for <codeph>IS NOT DISTINCT FROM</codeph>.
+        It is typically used as a <codeph>NULL</codeph>-safe equality operator in join queries.
+        That is, <codeph>A &lt;=&gt; B</codeph> is true if <codeph>A</codeph> equals <codeph>B</codeph>
+        or if both <codeph>A</codeph> and <codeph>B</codeph> are <codeph>NULL</codeph>.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p>
+        This operator provides concise notation for comparing two values and always producing a <codeph>true</codeph> or
+        <codeph>false</codeph> result, without treating <codeph>NULL</codeph> as a special case. Otherwise, to unambiguously distinguish
+        between two values requires a compound expression involving <codeph>IS [NOT] NULL</codeph> tests of both operands in addition to the
+        <codeph>=</codeph> or <codeph>!=</codeph> operator.
+      </p>
+
+      <p>
+        The <codeph>&lt;=&gt;</codeph> operator, used like an equality operator in a join query,
+        is more efficient than the equivalent clause: <codeph>A = B OR (A IS NULL AND B IS NULL)</codeph>.
+        The <codeph>&lt;=&gt;</codeph> operator can use a hash join, while the <codeph>OR</codeph> expression
+        cannot.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following examples show how <codeph>IS DISTINCT FROM</codeph> gives output similar to
+        the <codeph>!=</codeph> operator, and <codeph>IS NOT DISTINCT FROM</codeph> gives output
+        similar to the <codeph>=</codeph> operator. The exception is when the expression involves
+        a <codeph>NULL</codeph> value on one side or both sides, where <codeph>!=</codeph> and
+        <codeph>=</codeph> return <codeph>NULL</codeph> but the <codeph>IS [NOT] DISTINCT FROM</codeph>
+        operators still return <codeph>true</codeph> or <codeph>false</codeph>.
+      </p>
+
+<codeblock>
+select 1 is distinct from 0, 1 != 0;
++----------------------+--------+
+| 1 is distinct from 0 | 1 != 0 |
++----------------------+--------+
+| true                 | true   |
++----------------------+--------+
+
+select 1 is distinct from 1, 1 != 1;
++----------------------+--------+
+| 1 is distinct from 1 | 1 != 1 |
++----------------------+--------+
+| false                | false  |
++----------------------+--------+
+
+select 1 is distinct from null, 1 != null;
++-------------------------+-----------+
+| 1 is distinct from null | 1 != null |
++-------------------------+-----------+
+| true                    | NULL      |
++-------------------------+-----------+
+
+select null is distinct from null, null != null;
++----------------------------+--------------+
+| null is distinct from null | null != null |
++----------------------------+--------------+
+| false                      | NULL         |
++----------------------------+--------------+
+
+select 1 is not distinct from 0, 1 = 0;
++--------------------------+-------+
+| 1 is not distinct from 0 | 1 = 0 |
++--------------------------+-------+
+| false                    | false |
++--------------------------+-------+
+
+select 1 is not distinct from 1, 1 = 1;
++--------------------------+-------+
+| 1 is not distinct from 1 | 1 = 1 |
++--------------------------+-------+
+| true                     | true  |
++--------------------------+-------+
+
+select 1 is not distinct from null, 1 = null;
++-----------------------------+----------+
+| 1 is not distinct from null | 1 = null |
++-----------------------------+----------+
+| false                       | NULL     |
++-----------------------------+----------+
+
+select null is not distinct from null, null = null;
++--------------------------------+-------------+
+| null is not distinct from null | null = null |
++--------------------------------+-------------+
+| true                           | NULL        |
++--------------------------------+-------------+
+</codeblock>
+
+      <p>
+        The following example shows how <codeph>IS DISTINCT FROM</codeph> considers
+        <codeph>CHAR</codeph> values to be the same (not distinct from each other)
+        if they only differ in the number of trailing spaces. Therefore, sometimes
+        the result of an <codeph>IS [NOT] DISTINCT FROM</codeph> operator differs
+        depending on whether the values are <codeph>STRING</codeph>/<codeph>VARCHAR</codeph>
+        or <codeph>CHAR</codeph>.
+      </p>
+
+<codeblock>
+select
+  'x' is distinct from 'x ' as string_with_trailing_spaces,
+  cast('x' as char(5)) is distinct from cast('x ' as char(5)) as char_with_trailing_spaces;
++-----------------------------+---------------------------+
+| string_with_trailing_spaces | char_with_trailing_spaces |
++-----------------------------+---------------------------+
+| true                        | false                     |
++-----------------------------+---------------------------+
+</codeblock>
+    </conbody>
+
+  </concept>
+
+  <concept id="is_null">
+
+    <title>IS NULL Operator</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">IS NULL operator</indexterm>
+
+        <indexterm audience="Cloudera">IS NOT NULL operator</indexterm>
+        The <codeph>IS NULL</codeph> operator, and its converse the <codeph>IS NOT NULL</codeph> operator, test whether a specified value is
+        <codeph><xref href="impala_literals.xml#null">NULL</xref></codeph>. Because using <codeph>NULL</codeph> with any of the other
+        comparison operators such as <codeph>=</codeph> or <codeph>!=</codeph> also returns <codeph>NULL</codeph> rather than
+        <codeph>TRUE</codeph> or <codeph>FALSE</codeph>, you use a special-purpose comparison operator to check for this special condition.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock><varname>expression</varname> IS NULL
+<varname>expression</varname> IS NOT NULL
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p>
+        In many cases, <codeph>NULL</codeph> values indicate some incorrect or incomplete processing during data ingestion or conversion.
+        You might check whether any values in a column are <codeph>NULL</codeph>, and if so take some followup action to fill them in.
+      </p>
+
+      <p>
+        With sparse data, often represented in <q>wide</q> tables, it is common for most values to be <codeph>NULL</codeph> with only an
+        occasional non-<codeph>NULL</codeph> value. In those cases, you can use the <codeph>IS NOT NULL</codeph> operator to identify the
+        rows containing any data at all for a particular column, regardless of the actual value.
+      </p>
+
+      <p>
+        With a well-designed database schema, effective use of <codeph>NULL</codeph> values and <codeph>IS NULL</codeph> and <codeph>IS NOT
+        NULL</codeph> operators can save having to design custom logic around special values such as 0, -1, <codeph>'N/A'</codeph>, empty
+        string, and so on. <codeph>NULL</codeph> lets you distinguish between a value that is known to be 0, false, or empty, and a truly
+        unknown value.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+      <p rev="2.3.0">
+        This operator is not applicable to complex type columns (<codeph>STRUCT</codeph>,
+        <codeph>ARRAY</codeph>, or <codeph>MAP</codeph>). Using a complex type column
+        with <codeph>IS NULL</codeph> or <codeph>IS NOT NULL</codeph> causes a query error.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>-- If this value is non-zero, something is wrong.
+select count(*) from employees where employee_id is null;
+
+-- With data from disparate sources, some fields might be blank.
+-- Not necessarily an error condition.
+select count(*) from census where household_income is null;
+
+-- Sometimes we expect fields to be null, and followup action
+-- is needed when they are not.
+select count(*) from web_traffic where weird_http_code is not null;</codeblock>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="like">
+
+    <title>LIKE Operator</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">LIKE operator</indexterm>
+        A comparison operator for <codeph>STRING</codeph> data, with basic wildcard capability using the underscore
+        (<codeph>_</codeph>) to match a single character and the percent sign (<codeph>%</codeph>) to match multiple
+        characters. The argument expression must match the entire string value.
+        Typically, it is more efficient to put any <codeph>%</codeph> wildcard match at the end of the string.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock><varname>string_expression</varname> LIKE <varname>wildcard_expression</varname>
+<varname>string_expression</varname> NOT LIKE <varname>wildcard_expression</varname>
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_caveat_no_operator"/>
+
+<!-- To do: construct a LIKE example for complex types. -->
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>select distinct c_last_name from customer where c_last_name like 'Mc%' or c_last_name like 'Mac%';
+select count(c_last_name) from customer where c_last_name like 'M%';
+select c_email_address from customer where c_email_address like '%.edu';
+
+-- We can find 4-letter names beginning with 'M' by calling functions...
+select distinct c_last_name from customer where length(c_last_name) = 4 and substr(c_last_name,1,1) = 'M';
+-- ...or in a more readable way by matching M followed by exactly 3 characters.
+select distinct c_last_name from customer where c_last_name like 'M___';</codeblock>
+
+      <p rev="2.5.0">
+        For case-insensitive comparisons, see <xref href="impala_operators.xml#ilike"/>.
+        For a more general kind of search operator using regular expressions, see <xref href="impala_operators.xml#regexp"/>
+        or its case-insensitive counterpart <xref href="impala_operators.xml#iregexp"/>.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="logical_operators">
+
+    <title>Logical Operators</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">logical operators</indexterm>
+        Logical operators return a <codeph>BOOLEAN</codeph> value, based on a binary or unary logical operation between arguments that are
+        also Booleans. Typically, the argument expressions use <xref href="impala_operators.xml#comparison_operators">comparison
+        operators</xref>.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock><varname>boolean_expression</varname> <varname>binary_logical_operator</varname> <varname>boolean_expression</varname>
+<varname>unary_logical_operator</varname> <varname>boolean_expression</varname>
+</codeblock>
+
+      <p>
+        The Impala logical operators are:
+      </p>
+
+      <ul>
+        <li>
+          <codeph>AND</codeph>: A binary operator that returns <codeph>true</codeph> if its left-hand and right-hand arguments both evaluate
+          to <codeph>true</codeph>, <codeph>NULL</codeph> if either argument is <codeph>NULL</codeph>, and <codeph>false</codeph> otherwise.
+        </li>
+
+        <li>
+          <codeph>OR</codeph>: A binary operator that returns <codeph>true</codeph> if either of its left-hand and right-hand arguments
+          evaluate to <codeph>true</codeph>, <codeph>NULL</codeph> if one argument is <codeph>NULL</codeph> and the other is either
+          <codeph>NULL</codeph> or <codeph>false</codeph>, and <codeph>false</codeph> otherwise.
+        </li>
+
+        <li>
+          <codeph>NOT</codeph>: A unary operator that flips the state of a Boolean expression from <codeph>true</codeph> to
+          <codeph>false</codeph>, or <codeph>false</codeph> to <codeph>true</codeph>. If the argument expression is <codeph>NULL</codeph>,
+          the result remains <codeph>NULL</codeph>. (When <codeph>NOT</codeph> is used this way as a unary logical operator, it works
+          differently than the <codeph>IS NOT NULL</codeph> comparison operator, which returns <codeph>true</codeph> when applied to a
+          <codeph>NULL</codeph>.)
+        </li>
+      </ul>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_caveat_no_operator"/>
+
+      <p rev="2.3.0">
+        The following example shows how to do an arithmetic operation using a numeric field of a <codeph>STRUCT</codeph> type that is an
+        item within an <codeph>ARRAY</codeph> column. Once the scalar numeric value <codeph>R_NATIONKEY</codeph> is extracted, it can be
+        used in an arithmetic expression, such as multiplying by 10:
+      </p>
+
+<codeblock rev="2.3.0">
+-- The SMALLINT is a field within an array of structs.
+describe region;
++-------------+-------------------------+---------+
+| name        | type                    | comment |
++-------------+-------------------------+---------+
+| r_regionkey | smallint                |         |
+| r_name      | string                  |         |
+| r_comment   | string                  |         |
+| r_nations   | array&lt;struct&lt;           |         |
+|             |   n_nationkey:smallint, |         |
+|             |   n_name:string,        |         |
+|             |   n_comment:string      |         |
+|             | &gt;&gt;                      |         |
++-------------+-------------------------+---------+
+
+-- When we refer to the scalar value using dot notation,
+-- we can use arithmetic and comparison operators on it
+-- like any other number.
+select r_name, nation.item.n_name, nation.item.n_nationkey
+  from region, region.r_nations as nation
+where
+  nation.item.n_nationkey between 3 and 5
+  or nation.item.n_nationkey &lt; 15;
++-------------+----------------+------------------+
+| r_name      | item.n_name    | item.n_nationkey |
++-------------+----------------+------------------+
+| EUROPE      | UNITED KINGDOM | 23               |
+| EUROPE      | RUSSIA         | 22               |
+| EUROPE      | ROMANIA        | 19               |
+| ASIA        | VIETNAM        | 21               |
+| ASIA        | CHINA          | 18               |
+| AMERICA     | UNITED STATES  | 24               |
+| AMERICA     | PERU           | 17               |
+| AMERICA     | CANADA         | 3                |
+| MIDDLE EAST | SAUDI ARABIA   | 20               |
+| MIDDLE EAST | EGYPT          | 4                |
+| AFRICA      | MOZAMBIQUE     | 16               |
+| AFRICA      | ETHIOPIA       | 5                |
++-------------+----------------+------------------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        These examples demonstrate the <codeph>AND</codeph> operator:
+      </p>
+
+<codeblock>[localhost:21000] &gt; select true and true;
++---------------+
+| true and true |
++---------------+
+| true          |
++---------------+
+[localhost:21000] &gt; select true and false;
++----------------+
+| true and false |
++----------------+
+| false          |
++----------------+
+[localhost:21000] &gt; select false and false;
++-----------------+
+| false and false |
++-----------------+
+| false           |
++-----------------+
+[localhost:21000] &gt; select true and null;
++---------------+
+| true and null |
++---------------+
+| NULL          |
++---------------+
+[localhost:21000] &gt; select (10 &gt; 2) and (6 != 9);
++-----------------------+
+| (10 &gt; 2) and (6 != 9) |
++-----------------------+
+| true                  |
++-----------------------+
+</codeblock>
+
+      <p>
+        These examples demonstrate the <codeph>OR</codeph> operator:
+      </p>
+
+<codeblock>[localhost:21000] &gt; select true or true;
++--------------+
+| true or true |
++--------------+
+| true         |
++--------------+
+[localhost:21000] &gt; select true or false;
++---------------+
+| true or false |
++---------------+
+| true          |
++---------------+
+[localhost:21000] &gt; select false or false;
++----------------+
+| false or false |
++----------------+
+| false          |
++----------------+
+[localhost:21000] &gt; select true or null;
++--------------+
+| true or null |
++--------------+
+| true         |
++--------------+
+[localhost:21000] &gt; select null or true;
++--------------+
+| null or true |
++--------------+
+| true         |
++--------------+
+[localhost:21000] &gt; select false or null;
++---------------+
+| false or null |
++---------------+
+| NULL          |
++---------------+
+[localhost:21000] &gt; select (1 = 1) or ('hello' = 'world');
++--------------------------------+
+| (1 = 1) or ('hello' = 'world') |
++--------------------------------+
+| true                           |
++--------------------------------+
+[localhost:21000] &gt; select (2 + 2 != 4) or (-1 &gt; 0);
++--------------------------+
+| (2 + 2 != 4) or (-1 &gt; 0) |
++--------------------------+
+| false                    |
++--------------------------+
+</codeblock>
+
+      <p>
+        These examples demonstrate the <codeph>NOT</codeph> operator:
+      </p>
+
+<codeblock>[localhost:21000] &gt; select not true;
++----------+
+| not true |
++----------+
+| false    |
++----------+
+[localhost:21000] &gt; select not false;
++-----------+
+| not false |
++-----------+
+| true      |
++-----------+
+[localhost:21000] &gt; select not null;
++----------+
+| not null |
++----------+
+| NULL     |
++----------+
+[localhost:21000] &gt; select not (1=1);
++-------------+
+| not (1 = 1) |
++-------------+
+| false       |
++-------------+
+</codeblock>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="regexp">
+
+    <title>REGEXP Operator</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">REGEXP operator</indexterm>
+        Tests whether a value matches a regular expression. Uses the POSIX regular expression syntax where <codeph>^</codeph> and
+        <codeph>$</codeph> match the beginning and end of the string, <codeph>.</codeph> represents any single character, <codeph>*</codeph>
+        represents a sequence of zero or more items, <codeph>+</codeph> represents a sequence of one or more items, <codeph>?</codeph>
+        produces a non-greedy match, and so on.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock><varname>string_expression</varname> REGEXP <varname>regular_expression</varname>
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/regular_expression_whole_string"/>
+
+      <p>
+        The <codeph>RLIKE</codeph> operator is a synonym for <codeph>REGEXP</codeph>.
+      </p>
+
+      <p>
+        The <codeph>|</codeph> symbol is the alternation operator, typically used within <codeph>()</codeph> to match different sequences.
+        The <codeph>()</codeph> groups do not allow backreferences. To retrieve the part of a value matched within a <codeph>()</codeph>
+        section, use the <codeph><xref href="impala_string_functions.xml#string_functions/regexp_extract">regexp_extract()</xref></codeph>
+        built-in function.
+      </p>
+
+      <note rev="1.3.1">
+          <p rev="1.3.1" conref="../shared/impala_common.xml#common/regexp_matching"/>
+      </note>
+
+      <p conref="../shared/impala_common.xml#common/regexp_re2"/>
+
+      <p conref="../shared/impala_common.xml#common/regexp_re2_warning"/>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_caveat_no_operator"/>
+
+<!-- To do: construct a REGEXP example for complex types. -->
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following examples demonstrate the identical syntax for the <codeph>REGEXP</codeph> and <codeph>RLIKE</codeph> operators.
+      </p>
+
+<!-- Same examples shown for both REGEXP and RLIKE operators. -->
+
+<codeblock conref="../shared/impala_common.xml#common/regexp_rlike_examples"/>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p rev="2.5.0">
+        For regular expression matching with case-insensitive comparisons, see <xref href="impala_operators.xml#iregexp"/>.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="rlike">
+
+    <title>RLIKE Operator</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">RLIKE operator</indexterm>
+        Synonym for the <codeph>REGEXP</codeph> operator. See <xref href="impala_operators.xml#regexp"/> for details.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following examples demonstrate the identical syntax for the <codeph>REGEXP</codeph> and <codeph>RLIKE</codeph> operators.
+      </p>
+
+<!-- Same examples shown for both REGEXP and RLIKE operators. -->
+
+<codeblock conref="../shared/impala_common.xml#common/regexp_rlike_examples"/>
+
+    </conbody>
+
+  </concept>
+
+</concept>


[27/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_float.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_float.xml b/docs/topics/impala_float.xml
new file mode 100644
index 0000000..8ef1144
--- /dev/null
+++ b/docs/topics/impala_float.xml
@@ -0,0 +1,94 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="float">
+
+  <title>FLOAT Data Type</title>
+  <titlealts audience="PDF"><navtitle>FLOAT</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      A single precision floating-point data type used in <codeph>CREATE TABLE</codeph> and <codeph>ALTER
+      TABLE</codeph> statements.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p>
+      In the column definition of a <codeph>CREATE TABLE</codeph> statement:
+    </p>
+
+<codeblock><varname>column_name</varname> FLOAT</codeblock>
+
+    <p>
+      <b>Range:</b> 1.40129846432481707e-45 .. 3.40282346638528860e+38, positive or negative
+    </p>
+
+    <p>
+      <b>Precision:</b> 6 to 9 significant digits, depending on usage. The number of significant digits does
+      not depend on the position of the decimal point.
+    </p>
+
+    <p>
+      <b>Representation:</b> The values are stored in 4 bytes, using
+      <xref href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format" scope="external" format="html">IEEE 754 Single Precision Binary Floating Point</xref> format.
+    </p>
+
+    <p>
+      <b>Conversions:</b> Impala automatically converts <codeph>FLOAT</codeph> to more precise
+      <codeph>DOUBLE</codeph> values, but not the other way around. You can use <codeph>CAST()</codeph> to convert
+      <codeph>FLOAT</codeph> values to <codeph>TINYINT</codeph>, <codeph>SMALLINT</codeph>, <codeph>INT</codeph>,
+      <codeph>BIGINT</codeph>, <codeph>STRING</codeph>, <codeph>TIMESTAMP</codeph>, or <codeph>BOOLEAN</codeph>.
+      You can use exponential notation in <codeph>FLOAT</codeph> literals or when casting from
+      <codeph>STRING</codeph>, for example <codeph>1.0e6</codeph> to represent one million.
+      <ph conref="../shared/impala_common.xml#common/cast_int_to_timestamp"/>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>CREATE TABLE t1 (x FLOAT);
+SELECT CAST(1000.5 AS FLOAT);
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/partitioning_imprecise"/>
+
+    <p conref="../shared/impala_common.xml#common/hbase_ok"/>
+
+    <p conref="../shared/impala_common.xml#common/parquet_ok"/>
+
+    <p conref="../shared/impala_common.xml#common/text_bulky"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/compatibility_blurb"/> -->
+
+    <p conref="../shared/impala_common.xml#common/internals_4_bytes"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/added_in_20"/> -->
+
+    <p conref="../shared/impala_common.xml#common/column_stats_constant"/>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+<!-- This conref appears under SUM(), AVG(), FLOAT, and DOUBLE topics. -->
+
+    <p conref="../shared/impala_common.xml#common/sum_double"/>
+
+    <p conref="../shared/impala_common.xml#common/float_double_decimal_caveat"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_literals.xml#numeric_literals"/>, <xref href="impala_math_functions.xml#math_functions"/>,
+      <xref href="impala_double.xml#double"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_functions.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_functions.xml b/docs/topics/impala_functions.xml
new file mode 100644
index 0000000..55a36dc
--- /dev/null
+++ b/docs/topics/impala_functions.xml
@@ -0,0 +1,162 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="builtins">
+
+  <title id="title_functions">Impala Built-In Functions</title>
+  <titlealts audience="PDF"><navtitle>Built-In Functions</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <!-- To do:
+      Opportunity to conref some material between here and the "Functions" topic under "Schema Objects".
+    -->
+
+    <p>
+      Impala supports several categories of built-in functions. These functions let you perform mathematical
+      calculations, string manipulation, date calculations, and other kinds of data transformations directly in
+      <codeph>SELECT</codeph> statements. The built-in functions let a SQL query return results with all
+      formatting, calculating, and type conversions applied, rather than performing time-consuming postprocessing
+      in another application. By applying function calls where practical, you can make a SQL query that is as
+      convenient as an expression in a procedural programming language or a formula in a spreadsheet.
+    </p>
+
+    <p>
+      The categories of functions supported by Impala are:
+    </p>
+
+    <ul>
+      <li>
+        <xref href="impala_math_functions.xml#math_functions"/>
+      </li>
+
+      <li>
+        <xref href="impala_conversion_functions.xml#conversion_functions"/>
+      </li>
+
+      <li>
+        <xref href="impala_datetime_functions.xml#datetime_functions"/>
+      </li>
+
+      <li>
+        <xref href="impala_conditional_functions.xml#conditional_functions"/>
+      </li>
+
+      <li>
+        <xref href="impala_string_functions.xml#string_functions"/>
+      </li>
+
+      <li>
+        Aggregation functions, explained in <xref href="impala_aggregate_functions.xml#aggregate_functions"/>.
+      </li>
+    </ul>
+
+    <p>
+      You call any of these functions through the <codeph>SELECT</codeph> statement. For most functions, you can
+      omit the <codeph>FROM</codeph> clause and supply literal values for any required arguments:
+    </p>
+
+<codeblock>select abs(-1);
++---------+
+| abs(-1) |
++---------+
+| 1       |
++---------+
+
+select concat('The rain ', 'in Spain');
++---------------------------------+
+| concat('the rain ', 'in spain') |
++---------------------------------+
+| The rain in Spain               |
++---------------------------------+
+
+select power(2,5);
++-------------+
+| power(2, 5) |
++-------------+
+| 32          |
++-------------+
+</codeblock>
+
+    <p>
+      When you use a <codeph>FROM</codeph> clause and specify a column name as a function argument, the function is
+      applied for each item in the result set:
+    </p>
+
+<!-- TK: make real output for these; change the queries if necessary to use tables I already have. -->
+
+<codeblock>select concat('Country = ',country_code) from all_countries where population &gt; 100000000;
+select round(price) as dollar_value from product_catalog where price between 0.0 and 100.0;
+</codeblock>
+
+    <p>
+      Typically, if any argument to a built-in function is <codeph>NULL</codeph>, the result value is also
+      <codeph>NULL</codeph>:
+    </p>
+
+<codeblock>select cos(null);
++-----------+
+| cos(null) |
++-----------+
+| NULL      |
++-----------+
+
+select power(2,null);
++----------------+
+| power(2, null) |
++----------------+
+| NULL           |
++----------------+
+
+select concat('a',null,'b');
++------------------------+
+| concat('a', null, 'b') |
++------------------------+
+| NULL                   |
++------------------------+
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/aggr1"/>
+
+<codeblock conref="../shared/impala_common.xml#common/aggr2"/>
+
+    <p conref="../shared/impala_common.xml#common/aggr3"/>
+
+    <p>
+      Aggregate functions are a special category with different rules. These functions calculate a return value
+      across all the items in a result set, so they do require a <codeph>FROM</codeph> clause in the query:
+    </p>
+
+<!-- TK: make real output for these; change the queries if necessary to use tables I already have. -->
+
+<codeblock>select count(product_id) from product_catalog;
+select max(height), avg(height) from census_data where age &gt; 20;
+</codeblock>
+
+    <p>
+      Aggregate functions also ignore <codeph>NULL</codeph> values rather than returning a <codeph>NULL</codeph>
+      result. For example, if some rows have <codeph>NULL</codeph> for a particular column, those rows are ignored
+      when computing the AVG() for that column. Likewise, specifying <codeph>COUNT(col_name)</codeph> in a query
+      counts only those rows where <codeph>col_name</codeph> contains a non-<codeph>NULL</codeph> value.
+    </p>
+
+    <p rev="2.0.0">
+      Analytic functions are a variation on aggregate functions. Instead of returning a single value, or an
+      identical value for each group of rows, they can compute values that vary based on a <q>window</q> consisting
+      of other rows around them in the result set.
+    </p>
+
+    <p outputclass="toc"/>
+
+  </conbody>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_functions_overview.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_functions_overview.xml b/docs/topics/impala_functions_overview.xml
new file mode 100644
index 0000000..0e3973b
--- /dev/null
+++ b/docs/topics/impala_functions_overview.xml
@@ -0,0 +1,116 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="functions">
+
+  <title>Overview of Impala Functions</title>
+  <titlealts audience="PDF"><navtitle>Functions</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Functions let you apply arithmetic, string, or other computations and transformations to Impala data. You
+      typically use them in <codeph>SELECT</codeph> lists and <codeph>WHERE</codeph> clauses to filter and format
+      query results so that the result set is exactly what you want, with no further processing needed on the
+      application side.
+    </p>
+
+    <p>
+      Scalar functions return a single result for each input row. See <xref href="impala_functions.xml#builtins"/>.
+    </p>
+
+<codeblock>[localhost:21000] > select name, population from country where continent = 'North America' order by population desc limit 4;
+[localhost:21000] > select upper(name), population from country where continent = 'North America' order by population desc limit 4;
++-------------+------------+
+| upper(name) | population |
++-------------+------------+
+| USA         | 320000000  |
+| MEXICO      | 122000000  |
+| CANADA      | 25000000   |
+| GUATEMALA   | 16000000   |
++-------------+------------+
+</codeblock>
+    <p>
+      Aggregate functions combine the results from multiple rows:
+      either a single result for the entire table, or a separate result for each group of rows.
+      Aggregate functions are frequently used in combination with <codeph>GROUP BY</codeph>
+      and <codeph>HAVING</codeph> clauses in the <codeph>SELECT</codeph> statement.
+      See <xref href="impala_aggregate_functions.xml#aggregate_functions"/>.
+    </p>
+
+<codeblock>[localhost:21000] > select continent, <b>sum(population)</b> as howmany from country <b>group by continent</b> order by howmany desc;
++---------------+------------+
+| continent     | howmany    |
++---------------+------------+
+| Asia          | 4298723000 |
+| Africa        | 1110635000 |
+| Europe        | 742452000  |
+| North America | 565265000  |
+| South America | 406740000  |
+| Oceania       | 38304000   |
++---------------+------------+
+</codeblock>
+
+    <p>
+      User-defined functions (UDFs) let you code your own logic.  They can be either scalar or aggregate functions.
+      UDFs let you implement important business or scientific logic using high-performance code for Impala to automatically parallelize.
+      You can also use UDFs to implement convenience functions to simplify reporting or porting SQL from other database systems.
+      See <xref href="impala_udf.xml#udfs"/>.
+    </p>
+
+<codeblock>[localhost:21000] > select <b>rot13('Hello world!')</b> as 'Weak obfuscation';
++------------------+
+| weak obfuscation |
++------------------+
+| Uryyb jbeyq!     |
++------------------+
+[localhost:21000] > select <b>likelihood_of_new_subatomic_particle(sensor1, sensor2, sensor3)</b> as probability
+                  > from experimental_results group by experiment;
+</codeblock>
+
+    <p>
+      Each function is associated with a specific database. For example, if you issue a <codeph>USE somedb</codeph>
+      statement followed by <codeph>CREATE FUNCTION somefunc</codeph>, the new function is created in the
+      <codeph>somedb</codeph> database, and you could refer to it through the fully qualified name
+      <codeph>somedb.somefunc</codeph>. You could then issue another <codeph>USE</codeph> statement
+      and create a function with the same name in a different database.
+    </p>
+
+    <p>
+      Impala built-in functions are associated with a special database named <codeph>_impala_builtins</codeph>,
+      which lets you refer to them from any database without qualifying the name.
+    </p>
+
+<codeblock>[localhost:21000] > show databases;
++-------------------------+
+| name                    |
++-------------------------+
+| <b>_impala_builtins</b>        |
+| analytic_functions      |
+| avro_testing            |
+| data_file_size          |
+...
+[localhost:21000] > show functions in _impala_builtins like '*subs*';
++-------------+-----------------------------------+
+| return type | signature                         |
++-------------+-----------------------------------+
+| STRING      | substr(STRING, BIGINT)            |
+| STRING      | substr(STRING, BIGINT, BIGINT)    |
+| STRING      | substring(STRING, BIGINT)         |
+| STRING      | substring(STRING, BIGINT, BIGINT) |
++-------------+-----------------------------------+
+</codeblock>
+
+    <p>
+      <b>Related statements:</b> <xref href="impala_create_function.xml#create_function"/>,
+      <xref href="impala_drop_function.xml#drop_function"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_glossary.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_glossary.xml b/docs/topics/impala_glossary.xml
new file mode 100644
index 0000000..f713738
--- /dev/null
+++ b/docs/topics/impala_glossary.xml
@@ -0,0 +1,834 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="glossary">
+
+  <title>Impala Glossary</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <section id="glos_ddl">
+
+      <title>DDL</title>
+
+      <p>
+        A category of SQL statements involving administrative or metadata-related operations. Many of these
+        statements start with the keywords <codeph>CREATE</codeph>, <codeph>ALTER</codeph>, or
+        <codeph>DROP</codeph>. Contrast with DML.
+      </p>
+    </section>
+
+    <section id="glos_dml">
+
+      <title>DML</title>
+
+      <p>
+        A category of SQL statements that modify table data. Impala has a limited set of DML statements, primarily
+        the <codeph>INSERT</codeph> statement. Contrast with DDL.
+      </p>
+    </section>
+
+    <section id="glos_catalog">
+
+      <title>catalog</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_catalogd">
+
+      <title>catalogd</title>
+
+      <p>
+        A daemon.
+      </p>
+    </section>
+
+    <section id="glos_impalad">
+
+      <title>impalad</title>
+
+      <p>
+        A daemon.
+      </p>
+    </section>
+
+    <section id="glos_statestored">
+
+      <title>statestored</title>
+
+      <p>
+        A daemon.
+      </p>
+    </section>
+
+    <section id="glos_statestore">
+
+      <title>statestore</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_metastore">
+
+      <title>metastore</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_table">
+
+      <title>table</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_database">
+
+      <title>database</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_hive">
+
+      <title>Hive</title>
+
+      <p>
+        Its full trademarked name is Apache Hive.
+      </p>
+    </section>
+
+    <section id="glos_hbase">
+
+      <title>HBase</title>
+
+      <p>
+        Its full trademarked name is Apache HBase.
+      </p>
+    </section>
+
+    <section id="glos_coordinator_node">
+
+      <title>coordinator node</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_sql">
+
+      <title>SQL</title>
+
+      <p>
+        An acronym for Structured Query Language, the industry standard language for database operations.
+        Implemented by components such as Impala and Hive.
+      </p>
+    </section>
+
+    <section id="glos_avro">
+
+      <title>Avro</title>
+
+      <p>
+        A file format.
+      </p>
+    </section>
+
+    <section id="glos_text">
+
+      <title>text</title>
+
+      <p>
+        A file format.
+      </p>
+    </section>
+
+    <section id="glos_hdfs">
+
+      <title>HDFS</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_block">
+
+      <title>block</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_parquet">
+
+      <title>Parquet</title>
+
+      <p>
+        A file format.
+      </p>
+    </section>
+
+    <section id="glos_rcfile">
+
+      <title>RCFile</title>
+
+      <p>
+        A file format.
+      </p>
+    </section>
+
+    <section id="glos_sequencefile">
+
+      <title>SequenceFile</title>
+
+      <p>
+        A file format.
+      </p>
+    </section>
+
+    <section id="glos_snappy">
+
+      <title>Snappy</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_file_format">
+
+      <title>file format</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_compression">
+
+      <title>compression</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_codec">
+
+      <title>codec</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_lzo">
+
+      <title>LZO</title>
+
+      <p>
+        A compression codec.
+      </p>
+    </section>
+
+    <section id="glos_gzip">
+
+      <title>gzip</title>
+
+      <p>
+        A compression codec.
+      </p>
+    </section>
+
+    <section id="glos_bzip2">
+
+      <title>BZip2</title>
+
+      <p>
+        A compression codec.
+      </p>
+    </section>
+
+    <section id="glos_sentry">
+
+      <title/>
+
+      <p></p>
+    </section>
+
+    <section id="glos_authorization">
+
+      <title>authorization</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_authentication">
+
+      <title>authentication</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_delegation">
+
+      <title>delegation</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_skew">
+
+      <title>skew</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_profile">
+
+      <title>profile</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_query">
+
+      <title>query</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_external_table">
+
+      <title>external table</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_internal_table">
+
+      <title>internal table</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_view">
+
+      <title>view</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_function">
+
+      <title>function</title>
+
+      <p>
+        In Impala, a kind of database object that performs customized processing or business logic. The kinds of
+        functions include built-in functions, UDFs, and UDAs.
+      </p>
+    </section>
+
+    <section id="glos_udf">
+
+      <title>UDF</title>
+
+      <p>
+        Acronym for user-defined function. In Impala, a function that you write in C++ or Java. A UDF (as opposed
+        to a UDA) returns a single scalar value for each row evaluated in the result set. On occasion, UDF is used
+        in a broad context to include all kinds of user-written functions, including UDAs.
+      </p>
+    </section>
+
+    <section id="glos_uda">
+
+      <title>UDA</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_aggregate_function">
+
+      <title>aggregate function</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_built_in_function">
+
+      <title>built-in function</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_analytic_function">
+
+      <title>analytic function</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_admission_control">
+
+      <title>admission control</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_llama">
+
+      <title>llama</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_impala">
+
+      <title>Impala</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_table_statistics">
+
+      <title>table statistics</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_column_statistics">
+
+      <title>column statistics</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_statistics">
+
+      <title>statistics</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_incremental_statistics">
+
+      <title>incremental statistics</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_hdfs_caching">
+
+      <title>HDFS caching</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_join_query">
+
+      <title>join query</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_straight_join">
+
+      <title>straight join</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_inner_join">
+
+      <title>inner join</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_outer_join">
+
+      <title>outer join</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_left_join">
+
+      <title>left join</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_right_join">
+
+      <title>right join</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_equijoin">
+
+      <title>equijoin</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_antijoin">
+
+      <title>antijoin</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_semijoin">
+
+      <title>semijoin</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_cross_join">
+
+      <title>cross join</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_hint">
+
+      <title>hint</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_distinct">
+
+      <title>distinct</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_impala_shell">
+
+      <title>impala-shell</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_odbc">
+
+      <title>ODBC</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_jdbc">
+
+      <title>JDBC</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_data_types">
+
+      <title>data types</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_performance">
+
+      <title>performance</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_scalability">
+
+      <title>scalability</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_hue">
+
+      <title>Hue</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_partition">
+
+      <title>partition</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_partitioning">
+
+      <title>partitioning</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_partitioned_table">
+
+      <title>partitioned table</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_hdfs_trashcan">
+
+      <title>HDFS trashcan</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_explain_plan">
+
+      <title>EXPLAIN plan</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_summary">
+
+      <title>summary</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_executive_summary">
+
+      <title>executive summary</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_alias">
+
+      <title>alias</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_identifier">
+
+      <title>identifier</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_query_option">
+
+      <title>query option</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_daemon">
+
+      <title>daemon</title>
+
+      <p>
+        The daemons that make up the Impala service are <cmdname>impalad</cmdname>, <cmdname>statestored</cmdname>,
+        and <cmdname>catalogd</cmdname>.
+      </p>
+    </section>
+
+    <section id="impala_service">
+
+      <title>Impala service</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_session">
+
+      <title>session</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_schema">
+
+      <title>schema</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_schema_evolution">
+
+      <title>schema evolution</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_literal">
+
+      <title>literal</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_string">
+
+      <title>string</title>
+
+      <p>
+        String values are represented in Impala by the data types <codeph>STRING</codeph>,
+        <codeph>VARCHAR</codeph>, and <codeph>CHAR</codeph>.
+      </p>
+    </section>
+
+    <section id="glos_number">
+
+      <title>number</title>
+
+      <p>
+        Numeric values are represented in Impala by the data types <codeph>INT</codeph>, <codeph>BIGINT</codeph>,
+        <codeph>SMALLINT</codeph>, <codeph>TINYINT</codeph>, <codeph>DECIMAL</codeph>, <codeph>DOUBLE</codeph>, and
+        <codeph>FLOAT</codeph>.
+      </p>
+    </section>
+
+    <section id="glos_date_time">
+
+      <title>date/time</title>
+
+      <p>
+        Date and time values are represented in Impala by the <codeph>TIMESTAMP</codeph> data type. Sometimes, for
+        convenience or as part of a partitioning plan, dates are represented as formatted <codeph>STRING</codeph>
+        columns or as separate integer columns for year, month, and day.
+      </p>
+    </section>
+
+    <section id="glos_cancel">
+
+      <title>cancel</title>
+
+      <p>
+        Halting an Impala query before it is completely finished.
+      </p>
+    </section>
+
+    <section id="glos_streaming">
+
+      <title>streaming</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_sorting">
+
+      <title>sorting</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_aggregation">
+
+      <title>aggregation</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_fragment">
+
+      <title>fragment</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_source">
+
+      <title>source</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_sink">
+
+      <title>sink</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_thrift">
+
+      <title>Thrift</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_delegation">
+
+      <title>delegation</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_authorization">
+
+      <title>authorization</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_authentication">
+
+      <title>authentication</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_partition_pruning">
+
+      <title>partition pruning</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_heartbeat">
+
+      <title>heartbeat</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_subscriber">
+
+      <title>subscriber</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_topic">
+
+      <title>topic</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_broadcast_join">
+
+      <title>broadcast join</title>
+
+      <p></p>
+    </section>
+
+    <section id="glos_shuffle_join">
+
+      <title>shuffle join</title>
+
+      <p></p>
+    </section>
+
+<!-- Use as a template for new definitions:
+    <section id="glos_">
+      <title></title>
+      <p>
+      </p>
+    </section>
+
+-->
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_grant.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_grant.xml b/docs/topics/impala_grant.xml
new file mode 100644
index 0000000..a14a717
--- /dev/null
+++ b/docs/topics/impala_grant.xml
@@ -0,0 +1,122 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="2.0.0" id="grant">
+
+  <title>GRANT Statement (<keyword keyref="impala20"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>GRANT</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="Sentry"/>
+      <data name="Category" value="Roles"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+      <!-- Consider whether to go deeper into categories like Security for the Sentry-related statements. -->
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.0.0">
+      <indexterm audience="Cloudera">GRANT statement</indexterm>
+<!-- Copied from Sentry docs. Turn into conref. I did some rewording for clarity. -->
+      The <codeph>GRANT</codeph> statement grants roles or privileges on specified objects to groups. Only Sentry
+      administrative users can grant roles to a group.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock rev="2.3.0 collevelauth">GRANT ROLE <varname>role_name</varname> TO GROUP <varname>group_name</varname>
+
+GRANT <varname>privilege</varname> ON <varname>object_type</varname> <varname>object_name</varname>
+   TO [ROLE] <varname>roleName</varname>
+   [WITH GRANT OPTION]
+
+<ph rev="2.3.0">privilege ::= SELECT | SELECT(<varname>column_name</varname>) | INSERT | ALL</ph>
+object_type ::= TABLE | DATABASE | SERVER | URI
+</codeblock>
+
+    <p>
+      Typically, the object name is an identifier. For URIs, it is a string literal.
+    </p>
+
+<!-- Turn privilege info into a conref or series of conrefs. (In both GRANT and REVOKE.) -->
+
+    <p conref="../shared/impala_common.xml#common/privileges_blurb"/>
+
+    <p>
+<!-- To do: The wording here can be fluid, and it's reused in several statements. Turn into a conref. -->
+      Only administrative users (initially, a predefined set of users specified in the Sentry service configuration
+      file) can use this statement.
+    </p>
+
+    <p>
+      The <codeph>WITH GRANT OPTION</codeph> clause allows members of the specified role to issue
+      <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements for those same privileges
+<!-- Copied from Sentry docs. Turn into conref. I did some rewording for clarity. -->
+      Hence, if a role has the <codeph>ALL</codeph> privilege on a database and the <codeph>WITH GRANT
+      OPTION</codeph> set, users granted that role can execute <codeph>GRANT</codeph>/<codeph>REVOKE</codeph>
+      statements only for that database or child tables of the database. This means a user could revoke the
+      privileges of the user that provided them the <codeph>GRANT OPTION</codeph>.
+    </p>
+
+    <p>
+<!-- Copied from Sentry docs. Turn into conref. Except I changed Hive to Impala. -->
+      Impala does not currently support revoking only the <codeph>WITH GRANT OPTION</codeph> from a privilege
+      previously granted to a role. To remove the <codeph>WITH GRANT OPTION</codeph>, revoke the privilege and
+      grant it again without the <codeph>WITH GRANT OPTION</codeph> flag.
+    </p>
+
+    <p rev="2.3.0 collevelauth">
+      The ability to grant or revoke <codeph>SELECT</codeph> privilege on specific columns is available
+      in <keyword keyref="impala23_full"/> and higher. See
+      <xref audience="integrated" href="sg_hive_sql.xml#concept_c2q_4qx_p4/col_level_auth_sentry"/><xref audience="standalone" href="https://www.cloudera.com/documentation/enterprise/latest/topics/sg_hive_sql.html" format="html" scope="external"/>
+      for details.
+    </p>
+
+<!-- Turn compatibility info into a conref or series of conrefs. (In both GRANT and REVOKE.) -->
+
+<!-- If they diverge during development, consider the version here in GRANT the authoritative one. -->
+
+    <p conref="../shared/impala_common.xml#common/compatibility_blurb"/>
+
+    <p>
+      <ul>
+        <li>
+          The Impala <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements are available in CDH 5.2 and
+          later.
+        </li>
+
+        <li>
+          In CDH 5.1 and later, Impala can make use of any roles and privileges specified by the
+          <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements in Hive, when your system is configured to
+          use the Sentry service instead of the file-based policy mechanism.
+        </li>
+
+        <li>
+          The Impala <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements for privileges do not require
+          the <codeph>ROLE</codeph> keyword to be repeated before each role name, unlike the equivalent Hive
+          statements.
+        </li>
+
+        <li conref="../shared/impala_common.xml#common/grant_revoke_single"/>
+      </ul>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_authorization.xml#authorization"/>, <xref href="impala_revoke.xml#revoke"/>,
+      <xref href="impala_create_role.xml#create_role"/>, <xref href="impala_drop_role.xml#drop_role"/>,
+      <xref href="impala_show.xml#show"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_group_by.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_group_by.xml b/docs/topics/impala_group_by.xml
new file mode 100644
index 0000000..2c6ade3
--- /dev/null
+++ b/docs/topics/impala_group_by.xml
@@ -0,0 +1,139 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="group_by">
+
+  <title>GROUP BY Clause</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Aggregate Functions"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Specify the <codeph>GROUP BY</codeph> clause in queries that use aggregation functions, such as
+      <codeph><xref href="impala_count.xml#count">COUNT()</xref></codeph>,
+      <codeph><xref href="impala_sum.xml#sum">SUM()</xref></codeph>,
+      <codeph><xref href="impala_avg.xml#avg">AVG()</xref></codeph>,
+      <codeph><xref href="impala_min.xml#min">MIN()</xref></codeph>, and
+      <codeph><xref href="impala_max.xml#max">MAX()</xref></codeph>. Specify in the
+      <codeph><xref href="impala_group_by.xml#group_by">GROUP BY</xref></codeph> clause the names of all the
+      columns that do not participate in the aggregation operation.
+    </p>
+
+    <!-- Good to show an example of cases where ORDER BY does and doesn't work with complex types. -->
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+    <p rev="2.3.0">
+      In <keyword keyref="impala23_full"/> and higher, the complex data types <codeph>STRUCT</codeph>,
+      <codeph>ARRAY</codeph>, and <codeph>MAP</codeph> are available. These columns cannot
+      be referenced directly in the <codeph>ORDER BY</codeph> clause.
+      When you query a complex type column, you use join notation to <q>unpack</q> the elements
+      of the complex type, and within the join query you can include an <codeph>ORDER BY</codeph>
+      clause to control the order in the result set of the scalar elements from the complex type.
+      See <xref href="impala_complex_types.xml#complex_types"/> for details about Impala support for complex types.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/zero_length_strings"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      For example, the following query finds the 5 items that sold the highest total quantity (using the
+      <codeph>SUM()</codeph> function, and also counts the number of sales transactions for those items (using the
+      <codeph>COUNT()</codeph> function). Because the column representing the item IDs is not used in any
+      aggregation functions, we specify that column in the <codeph>GROUP BY</codeph> clause.
+    </p>
+
+<codeblock>select
+  <b>ss_item_sk</b> as Item,
+  <b>count</b>(ss_item_sk) as Times_Purchased,
+  <b>sum</b>(ss_quantity) as Total_Quantity_Purchased
+from store_sales
+  <b>group by ss_item_sk</b>
+  order by sum(ss_quantity) desc
+  limit 5;
++-------+-----------------+--------------------------+
+| item  | times_purchased | total_quantity_purchased |
++-------+-----------------+--------------------------+
+| 9325  | 372             | 19072                    |
+| 4279  | 357             | 18501                    |
+| 7507  | 371             | 18475                    |
+| 5953  | 369             | 18451                    |
+| 16753 | 375             | 18446                    |
++-------+-----------------+--------------------------+</codeblock>
+
+    <p>
+      The <codeph>HAVING</codeph> clause lets you filter the results of aggregate functions, because you cannot
+      refer to those expressions in the <codeph>WHERE</codeph> clause. For example, to find the 5 lowest-selling
+      items that were included in at least 100 sales transactions, we could use this query:
+    </p>
+
+<codeblock>select
+  <b>ss_item_sk</b> as Item,
+  <b>count</b>(ss_item_sk) as Times_Purchased,
+  <b>sum</b>(ss_quantity) as Total_Quantity_Purchased
+from store_sales
+  <b>group by ss_item_sk</b>
+  <b>having times_purchased &gt;= 100</b>
+  order by sum(ss_quantity)
+  limit 5;
++-------+-----------------+--------------------------+
+| item  | times_purchased | total_quantity_purchased |
++-------+-----------------+--------------------------+
+| 13943 | 105             | 4087                     |
+| 2992  | 101             | 4176                     |
+| 4773  | 107             | 4204                     |
+| 14350 | 103             | 4260                     |
+| 11956 | 102             | 4275                     |
++-------+-----------------+--------------------------+</codeblock>
+
+    <p>
+      When performing calculations involving scientific or financial data, remember that columns with type
+      <codeph>FLOAT</codeph> or <codeph>DOUBLE</codeph> are stored as true floating-point numbers, which cannot
+      precisely represent every possible fractional value. Thus, if you include a <codeph>FLOAT</codeph> or
+      <codeph>DOUBLE</codeph> column in a <codeph>GROUP BY</codeph> clause, the results might not precisely match
+      literal values in your query or from an original Text data file. Use rounding operations, the
+      <codeph>BETWEEN</codeph> operator, or another arithmetic technique to match floating-point values that are
+      <q>near</q> literal values you expect. For example, this query on the <codeph>ss_wholesale_cost</codeph>
+      column returns cost values that are close but not identical to the original figures that were entered as
+      decimal fractions.
+    </p>
+
+<codeblock>select ss_wholesale_cost, avg(ss_quantity * ss_sales_price) as avg_revenue_per_sale
+  from sales
+  group by ss_wholesale_cost
+  order by avg_revenue_per_sale desc
+  limit 5;
++-------------------+----------------------+
+| ss_wholesale_cost | avg_revenue_per_sale |
++-------------------+----------------------+
+| 96.94000244140625 | 4454.351539300434    |
+| 95.93000030517578 | 4423.119941283189    |
+| 98.37999725341797 | 4332.516490316291    |
+| 97.97000122070312 | 4330.480601655014    |
+| 98.52999877929688 | 4291.316953108634    |
++-------------------+----------------------+</codeblock>
+
+    <p>
+      Notice how wholesale cost values originally entered as decimal fractions such as <codeph>96.94</codeph> and
+      <codeph>98.38</codeph> are slightly larger or smaller in the result set, due to precision limitations in the
+      hardware floating-point types. The imprecise representation of <codeph>FLOAT</codeph> and
+      <codeph>DOUBLE</codeph> values is why financial data processing systems often store currency using data types
+      that are less space-efficient but avoid these types of rounding errors.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_select.xml#select"/>,
+      <xref href="impala_aggregate_functions.xml#aggregate_functions"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_group_concat.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_group_concat.xml b/docs/topics/impala_group_concat.xml
new file mode 100644
index 0000000..0971875
--- /dev/null
+++ b/docs/topics/impala_group_concat.xml
@@ -0,0 +1,135 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.2" id="group_concat">
+
+  <title>GROUP_CONCAT Function</title>
+  <titlealts audience="PDF"><navtitle>GROUP_CONCAT</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="Aggregate Functions"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">group_concat() function</indexterm>
+      An aggregate function that returns a single string representing the argument value concatenated together for
+      each row of the result set. If the optional separator string is specified, the separator is added between
+      each pair of concatenated values. The default separator is a comma followed by a space.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<!-- Might allow DISTINCT at some point. Check: does it allow ALL now? -->
+
+<codeblock>GROUP_CONCAT([ALL] <varname>expression</varname> [, <varname>separator</varname>])</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/concat_blurb"/>
+
+    <p>
+      By default, returns a single string covering the whole result set. To include other columns or values in the
+      result set, or to produce multiple concatenated strings for subsets of rows, include a <codeph>GROUP
+      BY</codeph> clause in the query.
+    </p>
+
+    <p>
+      <b>Return type:</b> <codeph>STRING</codeph>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+    <p>
+      You cannot apply the <codeph>DISTINCT</codeph> operator to the argument of this function.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/analytic_not_allowed_caveat"/>
+
+    <p>
+      Currently, Impala returns an error if the result value grows larger than 1 GiB.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      The following examples illustrate various aspects of the <codeph>GROUP_CONCAT()</codeph> function.
+    </p>
+
+    <p>
+      You can call the function directly on a <codeph>STRING</codeph> column. To use it with a numeric column, cast
+      the value to <codeph>STRING</codeph>.
+    </p>
+
+<codeblock>[localhost:21000] &gt; create table t1 (x int, s string);
+[localhost:21000] &gt; insert into t1 values (1, "one"), (3, "three"), (2, "two"), (1, "one");
+[localhost:21000] &gt; select group_concat(s) from t1;
++----------------------+
+| group_concat(s)      |
++----------------------+
+| one, three, two, one |
++----------------------+
+[localhost:21000] &gt; select group_concat(cast(x as string)) from t1;
++---------------------------------+
+| group_concat(cast(x as string)) |
++---------------------------------+
+| 1, 3, 2, 1                      |
++---------------------------------+
+</codeblock>
+
+    <p>
+      The optional separator lets you format the result in flexible ways. The separator can be an arbitrary string
+      expression, not just a single character.
+    </p>
+
+<codeblock>[localhost:21000] &gt; select group_concat(s,"|") from t1;
++----------------------+
+| group_concat(s, '|') |
++----------------------+
+| one|three|two|one    |
++----------------------+
+[localhost:21000] &gt; select group_concat(s,'---') from t1;
++-------------------------+
+| group_concat(s, '---')  |
++-------------------------+
+| one---three---two---one |
++-------------------------+
+</codeblock>
+
+    <p>
+      The default separator is a comma followed by a space. To get a comma-delimited result without extra spaces,
+      specify a delimiter character that is only a comma.
+    </p>
+
+<codeblock>[localhost:21000] &gt; select group_concat(s,',') from t1;
++----------------------+
+| group_concat(s, ',') |
++----------------------+
+| one,three,two,one    |
++----------------------+
+</codeblock>
+
+    <p>
+      Including a <codeph>GROUP BY</codeph> clause lets you produce a different concatenated result for each group
+      in the result set. In this example, the only <codeph>X</codeph> value that occurs more than once is
+      <codeph>1</codeph>, so that is the only row in the result set where <codeph>GROUP_CONCAT()</codeph> returns a
+      delimited value. For groups containing a single value, <codeph>GROUP_CONCAT()</codeph> returns the original
+      value of its <codeph>STRING</codeph> argument.
+    </p>
+
+<codeblock>[localhost:21000] &gt; select x, group_concat(s) from t1 group by x;
++---+-----------------+
+| x | group_concat(s) |
++---+-----------------+
+| 2 | two             |
+| 3 | three           |
+| 1 | one, one        |
++---+-----------------+
+</codeblock>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_hadoop.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_hadoop.xml b/docs/topics/impala_hadoop.xml
new file mode 100644
index 0000000..7941a47
--- /dev/null
+++ b/docs/topics/impala_hadoop.xml
@@ -0,0 +1,165 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="intro_hadoop">
+
+  <title>How Impala Fits Into the Hadoop Ecosystem</title>
+  <titlealts audience="PDF"><navtitle>Role in the Hadoop Ecosystem</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Concepts"/>
+      <data name="Category" value="Hadoop"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Impala makes use of many familiar components within the Hadoop ecosystem. Impala can interchange data with
+      other Hadoop components, as both a consumer and a producer, so it can fit in flexible ways into your ETL and
+      ELT pipelines.
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="intro_hive">
+
+    <title>How Impala Works with Hive</title>
+
+    <conbody>
+
+      <p>
+        A major Impala goal is to make SQL-on-Hadoop operations fast and efficient enough to appeal to new
+        categories of users and open up Hadoop to new types of use cases. Where practical, it makes use of existing
+        Apache Hive infrastructure that many Hadoop users already have in place to perform long-running,
+        batch-oriented SQL queries.
+      </p>
+
+      <p>
+        In particular, Impala keeps its table definitions in a traditional MySQL or PostgreSQL database known as
+        the <b>metastore</b>, the same database where Hive keeps this type of data. Thus, Impala can access tables
+        defined or loaded by Hive, as long as all columns use Impala-supported data types, file formats, and
+        compression codecs.
+      </p>
+
+      <p>
+        The initial focus on query features and performance means that Impala can read more types of data with the
+        <codeph>SELECT</codeph> statement than it can write with the <codeph>INSERT</codeph> statement. To query
+        data using the Avro, RCFile, or SequenceFile <xref href="impala_file_formats.xml#file_formats">file
+        formats</xref>, you load the data using Hive.
+      </p>
+
+      <p rev="1.2.2">
+        The Impala query optimizer can also make use of <xref href="impala_perf_stats.xml#perf_table_stats">table
+        statistics</xref> and <xref href="impala_perf_stats.xml#perf_column_stats">column statistics</xref>.
+        Originally, you gathered this information with the <codeph>ANALYZE TABLE</codeph> statement in Hive; in
+        Impala 1.2.2 and higher, use the Impala <codeph><xref href="impala_compute_stats.xml#compute_stats">COMPUTE
+        STATS</xref></codeph> statement instead. <codeph>COMPUTE STATS</codeph> requires less setup, is more
+        reliable, and does not require switching back and forth between <cmdname>impala-shell</cmdname>
+        and the Hive shell.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="intro_metastore">
+
+    <title>Overview of Impala Metadata and the Metastore</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Concepts"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Hive"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        As discussed in <xref href="impala_hadoop.xml#intro_hive"/>, Impala maintains information about table
+        definitions in a central database known as the <b>metastore</b>. Impala also tracks other metadata for the
+        low-level characteristics of data files:
+      </p>
+
+      <ul>
+        <li>
+          The physical locations of blocks within HDFS.
+        </li>
+      </ul>
+
+      <p>
+        For tables with a large volume of data and/or many partitions, retrieving all the metadata for a table can
+        be time-consuming, taking minutes in some cases. Thus, each Impala node caches all of this metadata to
+        reuse for future queries against the same table.
+      </p>
+
+      <p rev="1.2">
+        If the table definition or the data in the table is updated, all other Impala daemons in the cluster must
+        receive the latest metadata, replacing the obsolete cached metadata, before issuing a query against that
+        table. In Impala 1.2 and higher, the metadata update is automatic, coordinated through the
+        <cmdname>catalogd</cmdname> daemon, for all DDL and DML statements issued through Impala. See
+        <xref href="impala_components.xml#intro_catalogd"/> for details.
+      </p>
+
+      <p>
+        For DDL and DML issued through Hive, or changes made manually to files in HDFS, you still use the
+        <codeph>REFRESH</codeph> statement (when new data files are added to existing tables) or the
+        <codeph>INVALIDATE METADATA</codeph> statement (for entirely new tables, or after dropping a table,
+        performing an HDFS rebalance operation, or deleting data files). Issuing <codeph>INVALIDATE
+        METADATA</codeph> by itself retrieves metadata for all the tables tracked by the metastore. If you know
+        that only specific tables have been changed outside of Impala, you can issue <codeph>REFRESH
+        <varname>table_name</varname></codeph> for each affected table to only retrieve the latest metadata for
+        those tables.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="intro_hdfs">
+
+    <title>How Impala Uses HDFS</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Concepts"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="HDFS"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        Impala uses the distributed filesystem HDFS as its primary data storage medium. Impala relies on the
+        redundancy provided by HDFS to guard against hardware or network outages on individual nodes. Impala table
+        data is physically represented as data files in HDFS, using familiar HDFS file formats and compression
+        codecs. When data files are present in the directory for a new table, Impala reads them all, regardless of
+        file name. New data is added in files with names controlled by Impala.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="intro_hbase">
+
+    <title>How Impala Uses HBase</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Concepts"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="HBase"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        HBase is an alternative to HDFS as a storage medium for Impala data. It is a database storage system built
+        on top of HDFS, without built-in SQL support. Many Hadoop users already have it configured and store large
+        (often sparse) data sets in it. By defining tables in Impala and mapping them to equivalent tables in
+        HBase, you can query the contents of the HBase tables through Impala, and even perform join queries
+        including both Impala and HBase tables. See <xref href="impala_hbase.xml#impala_hbase"/> for details.
+      </p>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_having.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_having.xml b/docs/topics/impala_having.xml
new file mode 100644
index 0000000..2de32bb
--- /dev/null
+++ b/docs/topics/impala_having.xml
@@ -0,0 +1,44 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="having">
+
+  <title>HAVING Clause</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Aggregate Functions"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Performs a filter operation on a <codeph>SELECT</codeph> query, by examining the results of aggregation
+      functions rather than testing each individual table row. Therefore, it is always used in conjunction with a
+      function such as <codeph><xref href="impala_count.xml#count">COUNT()</xref></codeph>,
+      <codeph><xref href="impala_sum.xml#sum">SUM()</xref></codeph>,
+      <codeph><xref href="impala_avg.xml#avg">AVG()</xref></codeph>,
+      <codeph><xref href="impala_min.xml#min">MIN()</xref></codeph>, or
+      <codeph><xref href="impala_max.xml#max">MAX()</xref></codeph>, and typically with the
+      <codeph><xref href="impala_group_by.xml#group_by">GROUP BY</xref></codeph> clause also.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+    <p rev="2.0.0">
+      The filter expression in the <codeph>HAVING</codeph> clause cannot include a scalar subquery.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_select.xml#select"/>,
+      <xref href="impala_group_by.xml#group_by"/>,
+      <xref href="impala_aggregate_functions.xml#aggregate_functions"/>
+    </p>
+
+  </conbody>
+</concept>


[31/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_explain_level.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_explain_level.xml b/docs/topics/impala_explain_level.xml
new file mode 100644
index 0000000..e0c30d2
--- /dev/null
+++ b/docs/topics/impala_explain_level.xml
@@ -0,0 +1,350 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.2" id="explain_level">
+
+  <title>EXPLAIN_LEVEL Query Option</title>
+  <titlealts audience="PDF"><navtitle>EXPLAIN_LEVEL</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Reports"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">EXPLAIN_LEVEL query option</indexterm>
+      Controls the amount of detail provided in the output of the <codeph>EXPLAIN</codeph> statement. The basic
+      output can help you identify high-level performance issues such as scanning a higher volume of data or more
+      partitions than you expect. The higher levels of detail show how intermediate results flow between nodes and
+      how different SQL operations such as <codeph>ORDER BY</codeph>, <codeph>GROUP BY</codeph>, joins, and
+      <codeph>WHERE</codeph> clauses are implemented within a distributed query.
+    </p>
+
+    <p>
+      <b>Type:</b> <codeph>STRING</codeph> or <codeph>INT</codeph>
+    </p>
+
+    <p>
+      <b>Default:</b> <codeph>1</codeph>
+    </p>
+
+    <p>
+      <b>Arguments:</b>
+    </p>
+
+    <p>
+      The allowed range of numeric values for this option is 0 to 3:
+    </p>
+
+    <ul>
+      <li>
+        <codeph>0</codeph> or <codeph>MINIMAL</codeph>: A barebones list, one line per operation. Primarily useful
+        for checking the join order in very long queries where the regular <codeph>EXPLAIN</codeph> output is too
+        long to read easily.
+      </li>
+
+      <li>
+        <codeph>1</codeph> or <codeph>STANDARD</codeph>: The default level of detail, showing the logical way that
+        work is split up for the distributed query.
+      </li>
+
+      <li>
+        <codeph>2</codeph> or <codeph>EXTENDED</codeph>: Includes additional detail about how the query planner
+        uses statistics in its decision-making process, to understand how a query could be tuned by gathering
+        statistics, using query hints, adding or removing predicates, and so on.
+      </li>
+
+      <li>
+        <codeph>3</codeph> or <codeph>VERBOSE</codeph>: The maximum level of detail, showing how work is split up
+        within each node into <q>query fragments</q> that are connected in a pipeline. This extra detail is
+        primarily useful for low-level performance testing and tuning within Impala itself, rather than for
+        rewriting the SQL code at the user level.
+      </li>
+    </ul>
+
+    <note>
+      Prior to Impala 1.3, the allowed argument range for <codeph>EXPLAIN_LEVEL</codeph> was 0 to 1: level 0 had
+      the mnemonic <codeph>NORMAL</codeph>, and level 1 was <codeph>VERBOSE</codeph>. In Impala 1.3 and higher,
+      <codeph>NORMAL</codeph> is not a valid mnemonic value, and <codeph>VERBOSE</codeph> still applies to the
+      highest level of detail but now corresponds to level 3. You might need to adjust the values if you have any
+      older <codeph>impala-shell</codeph> script files that set the <codeph>EXPLAIN_LEVEL</codeph> query option.
+    </note>
+
+    <p>
+      Changing the value of this option controls the amount of detail in the output of the <codeph>EXPLAIN</codeph>
+      statement. The extended information from level 2 or 3 is especially useful during performance tuning, when
+      you need to confirm whether the work for the query is distributed the way you expect, particularly for the
+      most resource-intensive operations such as join queries against large tables, queries against tables with
+      large numbers of partitions, and insert operations for Parquet tables. The extended information also helps to
+      check estimated resource usage when you use the admission control or resource management features explained
+      in <xref href="impala_resource_management.xml#resource_management"/>. See
+      <xref href="impala_explain.xml#explain"/> for the syntax of the <codeph>EXPLAIN</codeph> statement, and
+      <xref href="impala_explain_plan.xml#perf_explain"/> for details about how to use the extended information.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      As always, read the <codeph>EXPLAIN</codeph> output from bottom to top. The lowest lines represent the
+      initial work of the query (scanning data files), the lines in the middle represent calculations done on each
+      node and how intermediate results are transmitted from one node to another, and the topmost lines represent
+      the final results being sent back to the coordinator node.
+    </p>
+
+    <p>
+      The numbers in the left column are generated internally during the initial planning phase and do not
+      represent the actual order of operations, so it is not significant if they appear out of order in the
+      <codeph>EXPLAIN</codeph> output.
+    </p>
+
+    <p>
+      At all <codeph>EXPLAIN</codeph> levels, the plan contains a warning if any tables in the query are missing
+      statistics. Use the <codeph>COMPUTE STATS</codeph> statement to gather statistics for each table and suppress
+      this warning. See <xref href="impala_perf_stats.xml#perf_stats"/> for details about how the statistics help
+      query performance.
+    </p>
+
+    <p>
+      The <codeph>PROFILE</codeph> command in <cmdname>impala-shell</cmdname> always starts with an explain plan
+      showing full detail, the same as with <codeph>EXPLAIN_LEVEL=3</codeph>. <ph rev="1.4.0">After the explain
+      plan comes the executive summary, the same output as produced by the <codeph>SUMMARY</codeph> command in
+      <cmdname>impala-shell</cmdname>.</ph>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      These examples use a trivial, empty table to illustrate how the essential aspects of query planning are shown
+      in <codeph>EXPLAIN</codeph> output:
+    </p>
+
+<codeblock>[localhost:21000] &gt; create table t1 (x int, s string);
+[localhost:21000] &gt; set explain_level=1;
+[localhost:21000] &gt; explain select count(*) from t1;
++------------------------------------------------------------------------+
+| Explain String                                                         |
++------------------------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=10.00MB VCores=1               |
+| WARNING: The following tables are missing relevant table and/or column |
+|   statistics.                                                          |
+| explain_plan.t1                                                        |
+|                                                                        |
+| 03:AGGREGATE [MERGE FINALIZE]                                          |
+| |  output: sum(count(*))                                               |
+| |                                                                      |
+| 02:EXCHANGE [PARTITION=UNPARTITIONED]                                  |
+| |                                                                      |
+| 01:AGGREGATE                                                           |
+| |  output: count(*)                                                    |
+| |                                                                      |
+| 00:SCAN HDFS [explain_plan.t1]                                         |
+|    partitions=1/1 size=0B                                              |
++------------------------------------------------------------------------+
+[localhost:21000] &gt; explain select * from t1;
++------------------------------------------------------------------------+
+| Explain String                                                         |
++------------------------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=-9223372036854775808B VCores=0 |
+| WARNING: The following tables are missing relevant table and/or column |
+|   statistics.                                                          |
+| explain_plan.t1                                                        |
+|                                                                        |
+| 01:EXCHANGE [PARTITION=UNPARTITIONED]                                  |
+| |                                                                      |
+| 00:SCAN HDFS [explain_plan.t1]                                         |
+|    partitions=1/1 size=0B                                              |
++------------------------------------------------------------------------+
+[localhost:21000] &gt; set explain_level=2;
+[localhost:21000] &gt; explain select * from t1;
++------------------------------------------------------------------------+
+| Explain String                                                         |
++------------------------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=-9223372036854775808B VCores=0 |
+| WARNING: The following tables are missing relevant table and/or column |
+|   statistics.                                                          |
+| explain_plan.t1                                                        |
+|                                                                        |
+| 01:EXCHANGE [PARTITION=UNPARTITIONED]                                  |
+| |  hosts=0 per-host-mem=unavailable                                    |
+| |  tuple-ids=0 row-size=19B cardinality=unavailable                    |
+| |                                                                      |
+| 00:SCAN HDFS [explain_plan.t1, PARTITION=RANDOM]                       |
+|    partitions=1/1 size=0B                                              |
+|    table stats: unavailable                                            |
+|    column stats: unavailable                                           |
+|    hosts=0 per-host-mem=0B                                             |
+|    tuple-ids=0 row-size=19B cardinality=unavailable                    |
++------------------------------------------------------------------------+
+[localhost:21000] &gt; set explain_level=3;
+[localhost:21000] &gt; explain select * from t1;
++------------------------------------------------------------------------+
+| Explain String                                                         |
++------------------------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=-9223372036854775808B VCores=0 |
+<b>| WARNING: The following tables are missing relevant table and/or column |</b>
+<b>|   statistics.                                                          |</b>
+<b>| explain_plan.t1                                                        |</b>
+|                                                                        |
+| F01:PLAN FRAGMENT [PARTITION=UNPARTITIONED]                            |
+|   01:EXCHANGE [PARTITION=UNPARTITIONED]                                |
+|      hosts=0 per-host-mem=unavailable                                  |
+|      tuple-ids=0 row-size=19B cardinality=unavailable                  |
+|                                                                        |
+| F00:PLAN FRAGMENT [PARTITION=RANDOM]                                   |
+|   DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=01, PARTITION=UNPARTITIONED] |
+|   00:SCAN HDFS [explain_plan.t1, PARTITION=RANDOM]                     |
+|      partitions=1/1 size=0B                                            |
+<b>|      table stats: unavailable                                          |</b>
+<b>|      column stats: unavailable                                         |</b>
+|      hosts=0 per-host-mem=0B                                           |
+|      tuple-ids=0 row-size=19B cardinality=unavailable                  |
++------------------------------------------------------------------------+
+</codeblock>
+
+    <p>
+      As the warning message demonstrates, most of the information needed for Impala to do efficient query
+      planning, and for you to understand the performance characteristics of the query, requires running the
+      <codeph>COMPUTE STATS</codeph> statement for the table:
+    </p>
+
+<codeblock>[localhost:21000] &gt; compute stats t1;
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 1 partition(s) and 2 column(s). |
++-----------------------------------------+
+[localhost:21000] &gt; explain select * from t1;
++------------------------------------------------------------------------+
+| Explain String                                                         |
++------------------------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=-9223372036854775808B VCores=0 |
+|                                                                        |
+| F01:PLAN FRAGMENT [PARTITION=UNPARTITIONED]                            |
+|   01:EXCHANGE [PARTITION=UNPARTITIONED]                                |
+|      hosts=0 per-host-mem=unavailable                                  |
+|      tuple-ids=0 row-size=20B cardinality=0                            |
+|                                                                        |
+| F00:PLAN FRAGMENT [PARTITION=RANDOM]                                   |
+|   DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=01, PARTITION=UNPARTITIONED] |
+|   00:SCAN HDFS [explain_plan.t1, PARTITION=RANDOM]                     |
+|      partitions=1/1 size=0B                                            |
+<b>|      table stats: 0 rows total                                         |</b>
+<b>|      column stats: all                                                 |</b>
+|      hosts=0 per-host-mem=0B                                           |
+|      tuple-ids=0 row-size=20B cardinality=0                            |
++------------------------------------------------------------------------+
+</codeblock>
+
+    <p>
+      Joins and other complicated, multi-part queries are the ones where you most commonly need to examine the
+      <codeph>EXPLAIN</codeph> output and customize the amount of detail in the output. This example shows the
+      default <codeph>EXPLAIN</codeph> output for a three-way join query, then the equivalent output with a
+      <codeph>[SHUFFLE]</codeph> hint to change the join mechanism between the first two tables from a broadcast
+      join to a shuffle join.
+    </p>
+
+<codeblock>[localhost:21000] &gt; set explain_level=1;
+[localhost:21000] &gt; explain select one.*, two.*, three.* from t1 one, t1 two, t1 three where one.x = two.x and two.x = three.x;
++---------------------------------------------------------+
+| Explain String                                          |
++---------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=4.00GB VCores=3 |
+|                                                         |
+| 07:EXCHANGE [PARTITION=UNPARTITIONED]                   |
+| |                                                       |
+<b>| 04:HASH JOIN [INNER JOIN, BROADCAST]                    |</b>
+| |  hash predicates: two.x = three.x                     |
+| |                                                       |
+<b>| |--06:EXCHANGE [BROADCAST]                              |</b>
+| |  |                                                    |
+| |  02:SCAN HDFS [explain_plan.t1 three]                 |
+| |     partitions=1/1 size=0B                            |
+| |                                                       |
+<b>| 03:HASH JOIN [INNER JOIN, BROADCAST]                    |</b>
+| |  hash predicates: one.x = two.x                       |
+| |                                                       |
+<b>| |--05:EXCHANGE [BROADCAST]                              |</b>
+| |  |                                                    |
+| |  01:SCAN HDFS [explain_plan.t1 two]                   |
+| |     partitions=1/1 size=0B                            |
+| |                                                       |
+| 00:SCAN HDFS [explain_plan.t1 one]                      |
+|    partitions=1/1 size=0B                               |
++---------------------------------------------------------+
+[localhost:21000] &gt; explain select one.*, two.*, three.*
+                  &gt; from t1 one join [shuffle] t1 two join t1 three
+                  &gt; where one.x = two.x and two.x = three.x;
++---------------------------------------------------------+
+| Explain String                                          |
++---------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=4.00GB VCores=3 |
+|                                                         |
+| 08:EXCHANGE [PARTITION=UNPARTITIONED]                   |
+| |                                                       |
+<b>| 04:HASH JOIN [INNER JOIN, BROADCAST]                    |</b>
+| |  hash predicates: two.x = three.x                     |
+| |                                                       |
+<b>| |--07:EXCHANGE [BROADCAST]                              |</b>
+| |  |                                                    |
+| |  02:SCAN HDFS [explain_plan.t1 three]                 |
+| |     partitions=1/1 size=0B                            |
+| |                                                       |
+<b>| 03:HASH JOIN [INNER JOIN, PARTITIONED]                  |</b>
+| |  hash predicates: one.x = two.x                       |
+| |                                                       |
+<b>| |--06:EXCHANGE [PARTITION=HASH(two.x)]                  |</b>
+| |  |                                                    |
+| |  01:SCAN HDFS [explain_plan.t1 two]                   |
+| |     partitions=1/1 size=0B                            |
+| |                                                       |
+<b>| 05:EXCHANGE [PARTITION=HASH(one.x)]                     |</b>
+| |                                                       |
+| 00:SCAN HDFS [explain_plan.t1 one]                      |
+|    partitions=1/1 size=0B                               |
++---------------------------------------------------------+
+</codeblock>
+
+    <p>
+      For a join involving many different tables, the default <codeph>EXPLAIN</codeph> output might stretch over
+      several pages, and the only details you care about might be the join order and the mechanism (broadcast or
+      shuffle) for joining each pair of tables. In that case, you might set <codeph>EXPLAIN_LEVEL</codeph> to its
+      lowest value of 0, to focus on just the join order and join mechanism for each stage. The following example
+      shows how the rows from the first and second joined tables are hashed and divided among the nodes of the
+      cluster for further filtering; then the entire contents of the third table are broadcast to all nodes for the
+      final stage of join processing.
+    </p>
+
+<codeblock>[localhost:21000] &gt; set explain_level=0;
+[localhost:21000] &gt; explain select one.*, two.*, three.*
+                  &gt; from t1 one join [shuffle] t1 two join t1 three
+                  &gt; where one.x = two.x and two.x = three.x;
++---------------------------------------------------------+
+| Explain String                                          |
++---------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=4.00GB VCores=3 |
+|                                                         |
+| 08:EXCHANGE [PARTITION=UNPARTITIONED]                   |
+<b>| 04:HASH JOIN [INNER JOIN, BROADCAST]                    |</b>
+<b>| |--07:EXCHANGE [BROADCAST]                              |</b>
+| |  02:SCAN HDFS [explain_plan.t1 three]                 |
+<b>| 03:HASH JOIN [INNER JOIN, PARTITIONED]                  |</b>
+<b>| |--06:EXCHANGE [PARTITION=HASH(two.x)]                  |</b>
+| |  01:SCAN HDFS [explain_plan.t1 two]                   |
+<b>| 05:EXCHANGE [PARTITION=HASH(one.x)]                     |</b>
+| 00:SCAN HDFS [explain_plan.t1 one]                      |
++---------------------------------------------------------+
+</codeblock>
+
+<!-- Consider adding a related info section to collect the xrefs earlier on this page. -->
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_explain_plan.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_explain_plan.xml b/docs/topics/impala_explain_plan.xml
new file mode 100644
index 0000000..4fd721f
--- /dev/null
+++ b/docs/topics/impala_explain_plan.xml
@@ -0,0 +1,568 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="explain_plan">
+
+  <title>Understanding Impala Query Performance - EXPLAIN Plans and Query Profiles</title>
+  <titlealts audience="PDF"><navtitle>EXPLAIN Plans and Query Profiles</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Reports"/>
+      <data name="Category" value="Concepts"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      To understand the high-level performance considerations for Impala queries, read the output of the
+      <codeph>EXPLAIN</codeph> statement for the query. You can get the <codeph>EXPLAIN</codeph> plan without
+      actually running the query itself.
+    </p>
+
+    <p rev="1.4.0">
+      For an overview of the physical performance characteristics for a query, issue the <codeph>SUMMARY</codeph>
+      statement in <cmdname>impala-shell</cmdname> immediately after executing a query. This condensed information
+      shows which phases of execution took the most time, and how the estimates for memory usage and number of rows
+      at each phase compare to the actual values.
+    </p>
+
+    <p>
+      To understand the detailed performance characteristics for a query, issue the <codeph>PROFILE</codeph>
+      statement in <cmdname>impala-shell</cmdname> immediately after executing a query. This low-level information
+      includes physical details about memory, CPU, I/O, and network usage, and thus is only available after the
+      query is actually run.
+    </p>
+
+    <p outputclass="toc inpage"/>
+
+    <p>
+      Also, see <xref href="impala_hbase.xml#hbase_performance"/>
+      and <xref href="impala_s3.xml#s3_performance"/>
+      for examples of interpreting
+      <codeph>EXPLAIN</codeph> plans for queries against HBase tables
+      <ph rev="2.2.0">and data stored in the Amazon Simple Storage System (S3)</ph>.
+    </p>
+  </conbody>
+
+  <concept id="perf_explain">
+
+    <title>Using the EXPLAIN Plan for Performance Tuning</title>
+
+    <conbody>
+
+      <p>
+        The <codeph><xref href="impala_explain.xml#explain">EXPLAIN</xref></codeph> statement gives you an outline
+        of the logical steps that a query will perform, such as how the work will be distributed among the nodes
+        and how intermediate results will be combined to produce the final result set. You can see these details
+        before actually running the query. You can use this information to check that the query will not operate in
+        some very unexpected or inefficient way.
+      </p>
+
+<!-- Turn into a conref in ciiu_langref too. Relocate to common.xml. -->
+
+<codeblock conref="impala_explain.xml#explain/explain_plan_simple"/>
+
+      <p conref="../shared/impala_common.xml#common/explain_interpret"/>
+
+      <p>
+        The <codeph>EXPLAIN</codeph> plan is also printed at the beginning of the query profile report described in
+        <xref href="#perf_profile"/>, for convenience in examining both the logical and physical aspects of the
+        query side-by-side.
+      </p>
+
+      <p rev="1.2">
+        The amount of detail displayed in the <codeph>EXPLAIN</codeph> output is controlled by the
+        <xref href="impala_explain_level.xml#explain_level">EXPLAIN_LEVEL</xref> query option. You typically
+        increase this setting from <codeph>normal</codeph> to <codeph>verbose</codeph> (or from <codeph>0</codeph>
+        to <codeph>1</codeph>) when doublechecking the presence of table and column statistics during performance
+        tuning, or when estimating query resource usage in conjunction with the resource management features in CDH
+        5.
+      </p>
+
+      <!-- To do:
+        This is a good place to have a few examples.
+      -->
+    </conbody>
+  </concept>
+
+  <concept id="perf_summary">
+
+    <title>Using the SUMMARY Report for Performance Tuning</title>
+
+    <conbody>
+
+      <p>
+        The <codeph><xref href="impala_shell_commands.xml#shell_commands">SUMMARY</xref></codeph> command within
+        the <cmdname>impala-shell</cmdname> interpreter gives you an easy-to-digest overview of the timings for the
+        different phases of execution for a query. Like the <codeph>EXPLAIN</codeph> plan, it is easy to see
+        potential performance bottlenecks. Like the <codeph>PROFILE</codeph> output, it is available after the
+        query is run and so displays actual timing numbers.
+      </p>
+
+      <p>
+        The <codeph>SUMMARY</codeph> report is also printed at the beginning of the query profile report described
+        in <xref href="#perf_profile"/>, for convenience in examining high-level and low-level aspects of the query
+        side-by-side.
+      </p>
+
+      <p>
+        For example, here is a query involving an aggregate function, on a single-node VM. The different stages of
+        the query and their timings are shown (rolled up for all nodes), along with estimated and actual values
+        used in planning the query. In this case, the <codeph>AVG()</codeph> function is computed for a subset of
+        data on each node (stage 01) and then the aggregated results from all nodes are combined at the end (stage
+        03). You can see which stages took the most time, and whether any estimates were substantially different
+        than the actual data distribution. (When examining the time values, be sure to consider the suffixes such
+        as <codeph>us</codeph> for microseconds and <codeph>ms</codeph> for milliseconds, rather than just looking
+        for the largest numbers.)
+      </p>
+
+<codeblock>[localhost:21000] &gt; select avg(ss_sales_price) from store_sales where ss_coupon_amt = 0;
++---------------------+
+| avg(ss_sales_price) |
++---------------------+
+| 37.80770926328327   |
++---------------------+
+[localhost:21000] &gt; summary;
++--------------+--------+----------+----------+-------+------------+----------+---------------+-----------------+
+| Operator     | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail          |
++--------------+--------+----------+----------+-------+------------+----------+---------------+-----------------+
+| 03:AGGREGATE | 1      | 1.03ms   | 1.03ms   | 1     | 1          | 48.00 KB | -1 B          | MERGE FINALIZE  |
+| 02:EXCHANGE  | 1      | 0ns      | 0ns      | 1     | 1          | 0 B      | -1 B          | UNPARTITIONED   |
+| 01:AGGREGATE | 1      | 30.79ms  | 30.79ms  | 1     | 1          | 80.00 KB | 10.00 MB      |                 |
+| 00:SCAN HDFS | 1      | 5.45s    | 5.45s    | 2.21M | -1         | 64.05 MB | 432.00 MB     | tpc.store_sales |
++--------------+--------+----------+----------+-------+------------+----------+---------------+-----------------+
+</codeblock>
+
+      <p>
+        Notice how the longest initial phase of the query is measured in seconds (s), while later phases working on
+        smaller intermediate results are measured in milliseconds (ms) or even nanoseconds (ns).
+      </p>
+
+      <p>
+        Here is an example from a more complicated query, as it would appear in the <codeph>PROFILE</codeph>
+        output:
+      </p>
+
+<!-- This example taken from: https://github.com/cloudera/Impala/commit/af85d3b518089b8840ddea4356947e40d1aca9bd -->
+
+<codeblock>Operator              #Hosts   Avg Time   Max Time    #Rows  Est. #Rows  Peak Mem  Est. Peak Mem  Detail
+------------------------------------------------------------------------------------------------------------------------
+09:MERGING-EXCHANGE        1   79.738us   79.738us        5           5         0        -1.00 B  UNPARTITIONED
+05:TOP-N                   3   84.693us   88.810us        5           5  12.00 KB       120.00 B
+04:AGGREGATE               3    5.263ms    6.432ms        5           5  44.00 KB       10.00 MB  MERGE FINALIZE
+08:AGGREGATE               3   16.659ms   27.444ms   52.52K     600.12K   3.20 MB       15.11 MB  MERGE
+07:EXCHANGE                3    2.644ms      5.1ms   52.52K     600.12K         0              0  HASH(o_orderpriority)
+03:AGGREGATE               3  342.913ms  966.291ms   52.52K     600.12K  10.80 MB       15.11 MB
+02:HASH JOIN               3    2s165ms    2s171ms  144.87K     600.12K  13.63 MB      941.01 KB  INNER JOIN, BROADCAST
+|--06:EXCHANGE             3    8.296ms    8.692ms   57.22K      15.00K         0              0  BROADCAST
+|  01:SCAN HDFS            2    1s412ms    1s978ms   57.22K      15.00K  24.21 MB      176.00 MB  tpch.orders o
+00:SCAN HDFS               3    8s032ms    8s558ms    3.79M     600.12K  32.29 MB      264.00 MB  tpch.lineitem l
+</codeblock>
+    </conbody>
+  </concept>
+
+  <concept id="perf_profile">
+
+    <title>Using the Query Profile for Performance Tuning</title>
+
+    <conbody>
+
+      <p>
+        The <codeph>PROFILE</codeph> statement, available in the <cmdname>impala-shell</cmdname> interpreter,
+        produces a detailed low-level report showing how the most recent query was executed. Unlike the
+        <codeph>EXPLAIN</codeph> plan described in <xref href="#perf_explain"/>, this information is only available
+        after the query has finished. It shows physical details such as the number of bytes read, maximum memory
+        usage, and so on for each node. You can use this information to determine if the query is I/O-bound or
+        CPU-bound, whether some network condition is imposing a bottleneck, whether a slowdown is affecting some
+        nodes but not others, and to check that recommended configuration settings such as short-circuit local
+        reads are in effect.
+      </p>
+
+      <p rev="CDH-29157">
+        By default, time values in the profile output reflect the wall-clock time taken by an operation.
+        For values denoting system time or user time, the measurement unit is reflected in the metric
+        name, such as <codeph>ScannerThreadsSysTime</codeph> or <codeph>ScannerThreadsUserTime</codeph>.
+        For example, a multi-threaded I/O operation might show a small figure for wall-clock time,
+        while the corresponding system time is larger, representing the sum of the CPU time taken by each thread.
+        Or a wall-clock time figure might be larger because it counts time spent waiting, while
+        the corresponding system and user time figures only measure the time while the operation
+        is actively using CPU cycles.
+      </p>
+
+      <p>
+        The <xref href="impala_explain_plan.xml#perf_explain"><codeph>EXPLAIN</codeph> plan</xref> is also printed
+        at the beginning of the query profile report, for convenience in examining both the logical and physical
+        aspects of the query side-by-side. The
+        <xref href="impala_explain_level.xml#explain_level">EXPLAIN_LEVEL</xref> query option also controls the
+        verbosity of the <codeph>EXPLAIN</codeph> output printed by the <codeph>PROFILE</codeph> command.
+      </p>
+
+      <!-- To do:
+        This is a good place to have a few more examples.
+      -->
+
+      <p>
+        Here is an example of a query profile, from a relatively straightforward query on a single-node
+        pseudo-distributed cluster to keep the output relatively brief.
+      </p>
+
+<codeblock>[localhost:21000] &gt; profile;
+Query Runtime Profile:
+Query (id=6540a03d4bee0691:4963d6269b210ebd):
+  Summary:
+    Session ID: ea4a197f1c7bf858:c74e66f72e3a33ba
+    Session Type: BEESWAX
+    Start Time: 2013-12-02 17:10:30.263067000
+    End Time: 2013-12-02 17:10:50.932044000
+    Query Type: QUERY
+    Query State: FINISHED
+    Query Status: OK
+    Impala Version: impalad version 1.2.1 RELEASE (build edb5af1bcad63d410bc5d47cc203df3a880e9324)
+    User: cloudera
+    Network Address: 127.0.0.1:49161
+    Default Db: stats_testing
+    Sql Statement: select t1.s, t2.s from t1 join t2 on (t1.id = t2.parent)
+    Plan:
+----------------
+Estimated Per-Host Requirements: Memory=2.09GB VCores=2
+
+PLAN FRAGMENT 0
+  PARTITION: UNPARTITIONED
+
+  4:EXCHANGE
+     cardinality: unavailable
+     per-host memory: unavailable
+     tuple ids: 0 1
+
+PLAN FRAGMENT 1
+  PARTITION: RANDOM
+
+  STREAM DATA SINK
+    EXCHANGE ID: 4
+    UNPARTITIONED
+
+  2:HASH JOIN
+  |  join op: INNER JOIN (BROADCAST)
+  |  hash predicates:
+  |    t1.id = t2.parent
+  |  cardinality: unavailable
+  |  per-host memory: 2.00GB
+  |  tuple ids: 0 1
+  |
+  |----3:EXCHANGE
+  |       cardinality: unavailable
+  |       per-host memory: 0B
+  |       tuple ids: 1
+  |
+  0:SCAN HDFS
+     table=stats_testing.t1 #partitions=1/1 size=33B
+     table stats: unavailable
+     column stats: unavailable
+     cardinality: unavailable
+     per-host memory: 32.00MB
+     tuple ids: 0
+
+PLAN FRAGMENT 2
+  PARTITION: RANDOM
+
+  STREAM DATA SINK
+    EXCHANGE ID: 3
+    UNPARTITIONED
+
+  1:SCAN HDFS
+     table=stats_testing.t2 #partitions=1/1 size=960.00KB
+     table stats: unavailable
+     column stats: unavailable
+     cardinality: unavailable
+     per-host memory: 96.00MB
+     tuple ids: 1
+----------------
+    Query Timeline: 20s670ms
+       - Start execution: 2.559ms (2.559ms)
+       - Planning finished: 23.587ms (21.27ms)
+       - Rows available: 666.199ms (642.612ms)
+       - First row fetched: 668.919ms (2.719ms)
+       - Unregister query: 20s668ms (20s000ms)
+  ImpalaServer:
+     - ClientFetchWaitTimer: 19s637ms
+     - RowMaterializationTimer: 167.121ms
+  Execution Profile 6540a03d4bee0691:4963d6269b210ebd:(Active: 837.815ms, % non-child: 0.00%)
+    Per Node Peak Memory Usage: impala-1.example.com:22000(7.42 MB)
+     - FinalizationTimer: 0ns
+    Coordinator Fragment:(Active: 195.198ms, % non-child: 0.00%)
+      MemoryUsage(500.0ms): 16.00 KB, 7.42 MB, 7.33 MB, 7.10 MB, 6.94 MB, 6.71 MB, 6.56 MB, 6.40 MB, 6.17 MB, 6.02 MB, 5.79 MB, 5.63 MB, 5.48 MB, 5.25 MB, 5.09 MB, 4.86 MB, 4.71 MB, 4.47 MB, 4.32 MB, 4.09 MB, 3.93 MB, 3.78 MB, 3.55 MB, 3.39 MB, 3.16 MB, 3.01 MB, 2.78 MB, 2.62 MB, 2.39 MB, 2.24 MB, 2.08 MB, 1.85 MB, 1.70 MB, 1.54 MB, 1.31 MB, 1.16 MB, 948.00 KB, 790.00 KB, 553.00 KB, 395.00 KB, 237.00 KB
+      ThreadUsage(500.0ms): 1
+       - AverageThreadTokens: 1.00
+       - PeakMemoryUsage: 7.42 MB
+       - PrepareTime: 36.144us
+       - RowsProduced: 98.30K (98304)
+       - TotalCpuTime: 20s449ms
+       - TotalNetworkWaitTime: 191.630ms
+       - TotalStorageWaitTime: 0ns
+      CodeGen:(Active: 150.679ms, % non-child: 77.19%)
+         - CodegenTime: 0ns
+         - CompileTime: 139.503ms
+         - LoadTime: 10.7ms
+         - ModuleFileSize: 95.27 KB
+      EXCHANGE_NODE (id=4):(Active: 194.858ms, % non-child: 99.83%)
+         - BytesReceived: 2.33 MB
+         - ConvertRowBatchTime: 2.732ms
+         - DataArrivalWaitTime: 191.118ms
+         - DeserializeRowBatchTimer: 14.943ms
+         - FirstBatchArrivalWaitTime: 191.117ms
+         - PeakMemoryUsage: 7.41 MB
+         - RowsReturned: 98.30K (98304)
+         - RowsReturnedRate: 504.49 K/sec
+         - SendersBlockedTimer: 0ns
+         - SendersBlockedTotalTimer(*): 0ns
+    Averaged Fragment 1:(Active: 442.360ms, % non-child: 0.00%)
+      split sizes:  min: 33.00 B, max: 33.00 B, avg: 33.00 B, stddev: 0.00
+      completion times: min:443.720ms  max:443.720ms  mean: 443.720ms  stddev:0ns
+      execution rates: min:74.00 B/sec  max:74.00 B/sec  mean:74.00 B/sec  stddev:0.00 /sec
+      num instances: 1
+       - AverageThreadTokens: 1.00
+       - PeakMemoryUsage: 6.06 MB
+       - PrepareTime: 7.291ms
+       - RowsProduced: 98.30K (98304)
+       - TotalCpuTime: 784.259ms
+       - TotalNetworkWaitTime: 388.818ms
+       - TotalStorageWaitTime: 3.934ms
+      CodeGen:(Active: 312.862ms, % non-child: 70.73%)
+         - CodegenTime: 2.669ms
+         - CompileTime: 302.467ms
+         - LoadTime: 9.231ms
+         - ModuleFileSize: 95.27 KB
+      DataStreamSender (dst_id=4):(Active: 80.63ms, % non-child: 18.10%)
+         - BytesSent: 2.33 MB
+         - NetworkThroughput(*): 35.89 MB/sec
+         - OverallThroughput: 29.06 MB/sec
+         - PeakMemoryUsage: 5.33 KB
+         - SerializeBatchTime: 26.487ms
+         - ThriftTransmitTime(*): 64.814ms
+         - UncompressedRowBatchSize: 6.66 MB
+      HASH_JOIN_NODE (id=2):(Active: 362.25ms, % non-child: 3.92%)
+         - BuildBuckets: 1.02K (1024)
+         - BuildRows: 98.30K (98304)
+         - BuildTime: 12.622ms
+         - LoadFactor: 0.00
+         - PeakMemoryUsage: 6.02 MB
+         - ProbeRows: 3
+         - ProbeTime: 3.579ms
+         - RowsReturned: 98.30K (98304)
+         - RowsReturnedRate: 271.54 K/sec
+        EXCHANGE_NODE (id=3):(Active: 344.680ms, % non-child: 77.92%)
+           - BytesReceived: 1.15 MB
+           - ConvertRowBatchTime: 2.792ms
+           - DataArrivalWaitTime: 339.936ms
+           - DeserializeRowBatchTimer: 9.910ms
+           - FirstBatchArrivalWaitTime: 199.474ms
+           - PeakMemoryUsage: 156.00 KB
+           - RowsReturned: 98.30K (98304)
+           - RowsReturnedRate: 285.20 K/sec
+           - SendersBlockedTimer: 0ns
+           - SendersBlockedTotalTimer(*): 0ns
+      HDFS_SCAN_NODE (id=0):(Active: 13.616us, % non-child: 0.00%)
+         - AverageHdfsReadThreadConcurrency: 0.00
+         - AverageScannerThreadConcurrency: 0.00
+         - BytesRead: 33.00 B
+         - BytesReadLocal: 33.00 B
+         - BytesReadShortCircuit: 33.00 B
+         - NumDisksAccessed: 1
+         - NumScannerThreadsStarted: 1
+         - PeakMemoryUsage: 46.00 KB
+         - PerReadThreadRawHdfsThroughput: 287.52 KB/sec
+         - RowsRead: 3
+         - RowsReturned: 3
+         - RowsReturnedRate: 220.33 K/sec
+         - ScanRangesComplete: 1
+         - ScannerThreadsInvoluntaryContextSwitches: 26
+         - ScannerThreadsTotalWallClockTime: 55.199ms
+           - DelimiterParseTime: 2.463us
+           - MaterializeTupleTime(*): 1.226us
+           - ScannerThreadsSysTime: 0ns
+           - ScannerThreadsUserTime: 42.993ms
+         - ScannerThreadsVoluntaryContextSwitches: 1
+         - TotalRawHdfsReadTime(*): 112.86us
+         - TotalReadThroughput: 0.00 /sec
+    Averaged Fragment 2:(Active: 190.120ms, % non-child: 0.00%)
+      split sizes:  min: 960.00 KB, max: 960.00 KB, avg: 960.00 KB, stddev: 0.00
+      completion times: min:191.736ms  max:191.736ms  mean: 191.736ms  stddev:0ns
+      execution rates: min:4.89 MB/sec  max:4.89 MB/sec  mean:4.89 MB/sec  stddev:0.00 /sec
+      num instances: 1
+       - AverageThreadTokens: 0.00
+       - PeakMemoryUsage: 906.33 KB
+       - PrepareTime: 3.67ms
+       - RowsProduced: 98.30K (98304)
+       - TotalCpuTime: 403.351ms
+       - TotalNetworkWaitTime: 34.999ms
+       - TotalStorageWaitTime: 108.675ms
+      CodeGen:(Active: 162.57ms, % non-child: 85.24%)
+         - CodegenTime: 3.133ms
+         - CompileTime: 148.316ms
+         - LoadTime: 12.317ms
+         - ModuleFileSize: 95.27 KB
+      DataStreamSender (dst_id=3):(Active: 70.620ms, % non-child: 37.14%)
+         - BytesSent: 1.15 MB
+         - NetworkThroughput(*): 23.30 MB/sec
+         - OverallThroughput: 16.23 MB/sec
+         - PeakMemoryUsage: 5.33 KB
+         - SerializeBatchTime: 22.69ms
+         - ThriftTransmitTime(*): 49.178ms
+         - UncompressedRowBatchSize: 3.28 MB
+      HDFS_SCAN_NODE (id=1):(Active: 118.839ms, % non-child: 62.51%)
+         - AverageHdfsReadThreadConcurrency: 0.00
+         - AverageScannerThreadConcurrency: 0.00
+         - BytesRead: 960.00 KB
+         - BytesReadLocal: 960.00 KB
+         - BytesReadShortCircuit: 960.00 KB
+         - NumDisksAccessed: 1
+         - NumScannerThreadsStarted: 1
+         - PeakMemoryUsage: 869.00 KB
+         - PerReadThreadRawHdfsThroughput: 130.21 MB/sec
+         - RowsRead: 98.30K (98304)
+         - RowsReturned: 98.30K (98304)
+         - RowsReturnedRate: 827.20 K/sec
+         - ScanRangesComplete: 15
+         - ScannerThreadsInvoluntaryContextSwitches: 34
+         - ScannerThreadsTotalWallClockTime: 189.774ms
+           - DelimiterParseTime: 15.703ms
+           - MaterializeTupleTime(*): 3.419ms
+           - ScannerThreadsSysTime: 1.999ms
+           - ScannerThreadsUserTime: 44.993ms
+         - ScannerThreadsVoluntaryContextSwitches: 118
+         - TotalRawHdfsReadTime(*): 7.199ms
+         - TotalReadThroughput: 0.00 /sec
+    Fragment 1:
+      Instance 6540a03d4bee0691:4963d6269b210ebf (host=impala-1.example.com:22000):(Active: 442.360ms, % non-child: 0.00%)
+        Hdfs split stats (&lt;volume id&gt;:&lt;# splits&gt;/&lt;split lengths&gt;): 0:1/33.00 B
+        MemoryUsage(500.0ms): 69.33 KB
+        ThreadUsage(500.0ms): 1
+         - AverageThreadTokens: 1.00
+         - PeakMemoryUsage: 6.06 MB
+         - PrepareTime: 7.291ms
+         - RowsProduced: 98.30K (98304)
+         - TotalCpuTime: 784.259ms
+         - TotalNetworkWaitTime: 388.818ms
+         - TotalStorageWaitTime: 3.934ms
+        CodeGen:(Active: 312.862ms, % non-child: 70.73%)
+           - CodegenTime: 2.669ms
+           - CompileTime: 302.467ms
+           - LoadTime: 9.231ms
+           - ModuleFileSize: 95.27 KB
+        DataStreamSender (dst_id=4):(Active: 80.63ms, % non-child: 18.10%)
+           - BytesSent: 2.33 MB
+           - NetworkThroughput(*): 35.89 MB/sec
+           - OverallThroughput: 29.06 MB/sec
+           - PeakMemoryUsage: 5.33 KB
+           - SerializeBatchTime: 26.487ms
+           - ThriftTransmitTime(*): 64.814ms
+           - UncompressedRowBatchSize: 6.66 MB
+        HASH_JOIN_NODE (id=2):(Active: 362.25ms, % non-child: 3.92%)
+          ExecOption: Build Side Codegen Enabled, Probe Side Codegen Enabled, Hash Table Built Asynchronously
+           - BuildBuckets: 1.02K (1024)
+           - BuildRows: 98.30K (98304)
+           - BuildTime: 12.622ms
+           - LoadFactor: 0.00
+           - PeakMemoryUsage: 6.02 MB
+           - ProbeRows: 3
+           - ProbeTime: 3.579ms
+           - RowsReturned: 98.30K (98304)
+           - RowsReturnedRate: 271.54 K/sec
+          EXCHANGE_NODE (id=3):(Active: 344.680ms, % non-child: 77.92%)
+             - BytesReceived: 1.15 MB
+             - ConvertRowBatchTime: 2.792ms
+             - DataArrivalWaitTime: 339.936ms
+             - DeserializeRowBatchTimer: 9.910ms
+             - FirstBatchArrivalWaitTime: 199.474ms
+             - PeakMemoryUsage: 156.00 KB
+             - RowsReturned: 98.30K (98304)
+             - RowsReturnedRate: 285.20 K/sec
+             - SendersBlockedTimer: 0ns
+             - SendersBlockedTotalTimer(*): 0ns
+        HDFS_SCAN_NODE (id=0):(Active: 13.616us, % non-child: 0.00%)
+          Hdfs split stats (&lt;volume id&gt;:&lt;# splits&gt;/&lt;split lengths&gt;): 0:1/33.00 B
+          Hdfs Read Thread Concurrency Bucket: 0:0% 1:0%
+          File Formats: TEXT/NONE:1
+          ExecOption: Codegen enabled: 1 out of 1
+           - AverageHdfsReadThreadConcurrency: 0.00
+           - AverageScannerThreadConcurrency: 0.00
+           - BytesRead: 33.00 B
+           - BytesReadLocal: 33.00 B
+           - BytesReadShortCircuit: 33.00 B
+           - NumDisksAccessed: 1
+           - NumScannerThreadsStarted: 1
+           - PeakMemoryUsage: 46.00 KB
+           - PerReadThreadRawHdfsThroughput: 287.52 KB/sec
+           - RowsRead: 3
+           - RowsReturned: 3
+           - RowsReturnedRate: 220.33 K/sec
+           - ScanRangesComplete: 1
+           - ScannerThreadsInvoluntaryContextSwitches: 26
+           - ScannerThreadsTotalWallClockTime: 55.199ms
+             - DelimiterParseTime: 2.463us
+             - MaterializeTupleTime(*): 1.226us
+             - ScannerThreadsSysTime: 0ns
+             - ScannerThreadsUserTime: 42.993ms
+           - ScannerThreadsVoluntaryContextSwitches: 1
+           - TotalRawHdfsReadTime(*): 112.86us
+           - TotalReadThroughput: 0.00 /sec
+    Fragment 2:
+      Instance 6540a03d4bee0691:4963d6269b210ec0 (host=impala-1.example.com:22000):(Active: 190.120ms, % non-child: 0.00%)
+        Hdfs split stats (&lt;volume id&gt;:&lt;# splits&gt;/&lt;split lengths&gt;): 0:15/960.00 KB
+         - AverageThreadTokens: 0.00
+         - PeakMemoryUsage: 906.33 KB
+         - PrepareTime: 3.67ms
+         - RowsProduced: 98.30K (98304)
+         - TotalCpuTime: 403.351ms
+         - TotalNetworkWaitTime: 34.999ms
+         - TotalStorageWaitTime: 108.675ms
+        CodeGen:(Active: 162.57ms, % non-child: 85.24%)
+           - CodegenTime: 3.133ms
+           - CompileTime: 148.316ms
+           - LoadTime: 12.317ms
+           - ModuleFileSize: 95.27 KB
+        DataStreamSender (dst_id=3):(Active: 70.620ms, % non-child: 37.14%)
+           - BytesSent: 1.15 MB
+           - NetworkThroughput(*): 23.30 MB/sec
+           - OverallThroughput: 16.23 MB/sec
+           - PeakMemoryUsage: 5.33 KB
+           - SerializeBatchTime: 22.69ms
+           - ThriftTransmitTime(*): 49.178ms
+           - UncompressedRowBatchSize: 3.28 MB
+        HDFS_SCAN_NODE (id=1):(Active: 118.839ms, % non-child: 62.51%)
+          Hdfs split stats (&lt;volume id&gt;:&lt;# splits&gt;/&lt;split lengths&gt;): 0:15/960.00 KB
+          Hdfs Read Thread Concurrency Bucket: 0:0% 1:0%
+          File Formats: TEXT/NONE:15
+          ExecOption: Codegen enabled: 15 out of 15
+           - AverageHdfsReadThreadConcurrency: 0.00
+           - AverageScannerThreadConcurrency: 0.00
+           - BytesRead: 960.00 KB
+           - BytesReadLocal: 960.00 KB
+           - BytesReadShortCircuit: 960.00 KB
+           - NumDisksAccessed: 1
+           - NumScannerThreadsStarted: 1
+           - PeakMemoryUsage: 869.00 KB
+           - PerReadThreadRawHdfsThroughput: 130.21 MB/sec
+           - RowsRead: 98.30K (98304)
+           - RowsReturned: 98.30K (98304)
+           - RowsReturnedRate: 827.20 K/sec
+           - ScanRangesComplete: 15
+           - ScannerThreadsInvoluntaryContextSwitches: 34
+           - ScannerThreadsTotalWallClockTime: 189.774ms
+             - DelimiterParseTime: 15.703ms
+             - MaterializeTupleTime(*): 3.419ms
+             - ScannerThreadsSysTime: 1.999ms
+             - ScannerThreadsUserTime: 44.993ms
+           - ScannerThreadsVoluntaryContextSwitches: 118
+           - TotalRawHdfsReadTime(*): 7.199ms
+           - TotalReadThroughput: 0.00 /sec</codeblock>
+    </conbody>
+  </concept>
+</concept>



[24/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_insert.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_insert.xml b/docs/topics/impala_insert.xml
new file mode 100644
index 0000000..02ad8c6
--- /dev/null
+++ b/docs/topics/impala_insert.xml
@@ -0,0 +1,689 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="insert">
+
+  <title>INSERT Statement</title>
+  <titlealts audience="PDF"><navtitle>INSERT</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="ETL"/>
+      <data name="Category" value="Ingest"/>
+      <data name="Category" value="DML"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="S3"/>
+      <!-- <data name="Category" value="Kudu"/> -->
+      <!-- This is such an important statement, think if there are more applicable categories. -->
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">INSERT statement</indexterm>
+      Impala supports inserting into tables and partitions that you create with the Impala <codeph>CREATE
+      TABLE</codeph> statement, or pre-defined tables and partitions created through Hive.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>[<varname>with_clause</varname>]
+INSERT { INTO | OVERWRITE } [TABLE] <varname>table_name</varname>
+  [(<varname>column_list</varname>)]
+  [ PARTITION (<varname>partition_clause</varname>)]
+{
+    [<varname>hint_clause</varname>] <varname>select_statement</varname>
+  | VALUES (<varname>value</varname> [, <varname>value</varname> ...]) [, (<varname>value</varname> [, <varname>value</varname> ...]) ...]
+}
+
+partition_clause ::= <varname>col_name</varname> [= <varname>constant</varname>] [, <varname>col_name</varname> [= <varname>constant</varname>] ...]
+
+hint_clause ::= [SHUFFLE] | [NOSHUFFLE]    (Note: the square brackets are part of the syntax.)
+</codeblock>
+
+    <p>
+      <b>Appending or replacing (INTO and OVERWRITE clauses):</b>
+    </p>
+
+    <p>
+      The <codeph>INSERT INTO</codeph> syntax appends data to a table. The existing data files are left as-is, and
+      the inserted data is put into one or more new data files.
+    </p>
+
+    <p>
+      The <codeph>INSERT OVERWRITE</codeph> syntax replaces the data in a table.
+<!-- What happens with INSERT OVERWRITE if the target is a single partition or multiple partitions? -->
+<!-- If that gets too detailed, cover later under "Partitioning Considerations". -->
+      Currently, the overwritten data files are deleted immediately; they do not go through the HDFS trash
+      mechanism.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+    <p rev="2.3.0">
+      The <codeph>INSERT</codeph> statement currently does not support writing data files
+      containing complex types (<codeph>ARRAY</codeph>, <codeph>STRUCT</codeph>, and <codeph>MAP</codeph>).
+      To prepare Parquet data for such tables, you generate the data files outside Impala and then
+      use <codeph>LOAD DATA</codeph> or <codeph>CREATE EXTERNAL TABLE</codeph> to associate those
+      data files with the table. Currently, such tables must use the Parquet file format.
+      See <xref href="impala_complex_types.xml#complex_types"/> for details about working with complex types.
+    </p>
+
+    <p rev="kudu">
+      <b>Ignoring duplicate partition keys for Kudu tables (IGNORE clause)</b>
+    </p>
+
+    <p rev="kudu">
+      Normally, an <codeph>INSERT</codeph> operation into a Kudu table fails if
+      it would result in duplicate partition key columns for any rows.
+      Specify <codeph>INSERT IGNORE <varname>rest_of_statement</varname></codeph> to
+      make the <codeph>INSERT</codeph> continue in this case. The rows that would
+      have duplicate partition key columns are not inserted.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      Impala currently supports:
+    </p>
+
+    <ul>
+      <li>
+        Copy data from another table using <codeph>SELECT</codeph> query. In Impala 1.2.1 and higher, you can
+        combine <codeph>CREATE TABLE</codeph> and <codeph>INSERT</codeph> operations into a single step with the
+        <codeph>CREATE TABLE AS SELECT</codeph> syntax, which bypasses the actual <codeph>INSERT</codeph> keyword.
+      </li>
+
+      <li>
+        An optional <xref href="impala_with.xml#with"><codeph>WITH</codeph> clause</xref> before the
+        <codeph>INSERT</codeph> keyword, to define a subquery referenced in the <codeph>SELECT</codeph> portion.
+      </li>
+
+      <li>
+        Create one or more new rows using constant expressions through <codeph>VALUES</codeph> clause. (The
+        <codeph>VALUES</codeph> clause was added in Impala 1.0.1.)
+      </li>
+
+      <li rev="1.1">
+        <p>
+          By default, the first column of each newly inserted row goes into the first column of the table, the
+          second column into the second column, and so on.
+        </p>
+        <p>
+          You can also specify the columns to be inserted, an arbitrarily ordered subset of the columns in the
+          destination table, by specifying a column list immediately after the name of the destination table. This
+          feature lets you adjust the inserted columns to match the layout of a <codeph>SELECT</codeph> statement,
+          rather than the other way around. (This feature was added in Impala 1.1.)
+        </p>
+        <p>
+          The number of columns mentioned in the column list (known as the <q>column permutation</q>) must match
+          the number of columns in the <codeph>SELECT</codeph> list or the <codeph>VALUES</codeph> tuples. The
+          order of columns in the column permutation can be different than in the underlying table, and the columns
+          of each input row are reordered to match. If the number of columns in the column permutation is less than
+          in the destination table, all unmentioned columns are set to <codeph>NULL</codeph>.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          For a partitioned table, the optional <codeph>PARTITION</codeph> clause identifies which partition or
+          partitions the new values go into. If a partition key column is given a constant value such as
+          <codeph>PARTITION (year=2012)</codeph> or <codeph>PARTITION (year=2012, month=2)</codeph>, all the
+          inserted rows use those same values for those partition key columns and you omit any corresponding
+          columns in the source table from the <codeph>SELECT</codeph> list. This form is known as <q>static
+          partitioning</q>.
+        </p>
+        <p>
+          If a partition key column is mentioned but not assigned a value, such as in <codeph>PARTITION (year,
+          region)</codeph> (both columns unassigned) or <codeph>PARTITION(year, region='CA')</codeph>
+          (<codeph>year</codeph> column unassigned), the unassigned columns are filled in with the final columns of
+          the <codeph>SELECT</codeph> list. In this case, the number of columns in the <codeph>SELECT</codeph> list
+          must equal the number of columns in the column permutation plus the number of partition key columns not
+          assigned a constant value. This form is known as <q>dynamic partitioning</q>.
+        </p>
+        <p>
+          See <xref href="impala_partitioning.xml#partition_static_dynamic"/> for examples and performance
+          characteristics of static and dynamic partitioned inserts.
+        </p>
+      </li>
+
+      <li rev="1.2.2">
+        An optional hint clause immediately before the <codeph>SELECT</codeph> keyword, to fine-tune the behavior
+        when doing an <codeph>INSERT ... SELECT</codeph> operation into partitioned Parquet tables. The hint
+        keywords are <codeph>[SHUFFLE]</codeph> and <codeph>[NOSHUFFLE]</codeph>, including the square brackets.
+        Inserting into partitioned Parquet tables can be a resource-intensive operation because it potentially
+        involves many files being written to HDFS simultaneously, and separate
+        <ph rev="parquet_block_size">large</ph> memory buffers being allocated to buffer the data for each
+        partition. For usage details, see <xref href="impala_parquet.xml#parquet_etl"/>.
+      </li>
+    </ul>
+
+    <note>
+      <ul>
+        <li>
+          Insert commands that partition or add files result in changes to Hive metadata. Because Impala uses Hive
+          metadata, such changes may necessitate a metadata refresh. For more information, see the
+          <xref href="impala_refresh.xml#refresh">REFRESH</xref> function.
+        </li>
+
+        <li>
+          Currently, Impala can only insert data into tables that use the text and Parquet formats. For other file
+          formats, insert the data using Hive and use Impala to query it.
+        </li>
+
+        <li>
+          As an alternative to the <codeph>INSERT</codeph> statement, if you have existing data files elsewhere in
+          HDFS, the <codeph>LOAD DATA</codeph> statement can move those files into a table. This statement works
+          with tables of any file format.
+        </li>
+      </ul>
+    </note>
+
+    <p conref="../shared/impala_common.xml#common/dml_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      When you insert the results of an expression, particularly of a built-in function call, into a small numeric
+      column such as <codeph>INT</codeph>, <codeph>SMALLINT</codeph>, <codeph>TINYINT</codeph>, or
+      <codeph>FLOAT</codeph>, you might need to use a <codeph>CAST()</codeph> expression to coerce values into the
+      appropriate type. Impala does not automatically convert from a larger type to a smaller one. For example, to
+      insert cosine values into a <codeph>FLOAT</codeph> column, write <codeph>CAST(COS(angle) AS FLOAT)</codeph>
+      in the <codeph>INSERT</codeph> statement to make the conversion explicit.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/file_format_blurb"/>
+
+    <p rev="DOCS-1523">
+      Because Impala can read certain file formats that it cannot write,
+      the <codeph>INSERT</codeph> statement does not work for all kinds of
+      Impala tables. See <xref href="impala_file_formats.xml#file_formats"/>
+      for details about what file formats are supported by the
+      <codeph>INSERT</codeph> statement.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/insert_parquet_blocksize"/>
+
+    <p conref="../shared/impala_common.xml#common/sync_ddl_blurb"/>
+
+    <note conref="../shared/impala_common.xml#common/compute_stats_next"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      The following example sets up new tables with the same definition as the <codeph>TAB1</codeph> table from the
+      <xref href="impala_tutorial.xml#tutorial">Tutorial</xref> section, using different file
+      formats, and demonstrates inserting data into the tables created with the <codeph>STORED AS TEXTFILE</codeph>
+      and <codeph>STORED AS PARQUET</codeph> clauses:
+    </p>
+
+<codeblock>CREATE DATABASE IF NOT EXISTS file_formats;
+USE file_formats;
+
+DROP TABLE IF EXISTS text_table;
+CREATE TABLE text_table
+( id INT, col_1 BOOLEAN, col_2 DOUBLE, col_3 TIMESTAMP )
+STORED AS TEXTFILE;
+
+DROP TABLE IF EXISTS parquet_table;
+CREATE TABLE parquet_table
+( id INT, col_1 BOOLEAN, col_2 DOUBLE, col_3 TIMESTAMP )
+STORED AS PARQUET;</codeblock>
+
+    <p>
+      With the <codeph>INSERT INTO TABLE</codeph> syntax, each new set of inserted rows is appended to any existing
+      data in the table. This is how you would record small amounts of data that arrive continuously, or ingest new
+      batches of data alongside the existing data. For example, after running 2 <codeph>INSERT INTO TABLE</codeph>
+      statements with 5 rows each, the table contains 10 rows total:
+    </p>
+
+<codeblock>[localhost:21000] &gt; insert into table text_table select * from default.tab1;
+Inserted 5 rows in 0.41s
+
+[localhost:21000] &gt; insert into table text_table select * from default.tab1;
+Inserted 5 rows in 0.46s
+
+[localhost:21000] &gt; select count(*) from text_table;
++----------+
+| count(*) |
++----------+
+| 10       |
++----------+
+Returned 1 row(s) in 0.26s</codeblock>
+
+    <p>
+      With the <codeph>INSERT OVERWRITE TABLE</codeph> syntax, each new set of inserted rows replaces any existing
+      data in the table. This is how you load data to query in a data warehousing scenario where you analyze just
+      the data for a particular day, quarter, and so on, discarding the previous data each time. You might keep the
+      entire set of data in one raw table, and transfer and transform certain rows into a more compact and
+      efficient form to perform intensive analysis on that subset.
+    </p>
+
+    <p>
+      For example, here we insert 5 rows into a table using the <codeph>INSERT INTO</codeph> clause, then replace
+      the data by inserting 3 rows with the <codeph>INSERT OVERWRITE</codeph> clause. Afterward, the table only
+      contains the 3 rows from the final <codeph>INSERT</codeph> statement.
+    </p>
+
+<codeblock>[localhost:21000] &gt; insert into table parquet_table select * from default.tab1;
+Inserted 5 rows in 0.35s
+
+[localhost:21000] &gt; insert overwrite table parquet_table select * from default.tab1 limit 3;
+Inserted 3 rows in 0.43s
+[localhost:21000] &gt; select count(*) from parquet_table;
++----------+
+| count(*) |
++----------+
+| 3        |
++----------+
+Returned 1 row(s) in 0.43s</codeblock>
+
+    <p>
+      The <codeph><xref href="impala_insert.xml#values">VALUES</xref></codeph> clause lets you insert one or more
+      rows by specifying constant values for all the columns. The number, types, and order of the expressions must
+      match the table definition.
+    </p>
+
+    <note id="insert_values_warning">
+      The <codeph>INSERT ... VALUES</codeph> technique is not suitable for loading large quantities of data into
+      HDFS-based tables, because the insert operations cannot be parallelized, and each one produces a separate
+      data file. Use it for setting up small dimension tables or tiny amounts of data for experimenting with SQL
+      syntax, or with HBase tables. Do not use it for large ETL jobs or benchmark tests for load operations. Do not
+      run scripts with thousands of <codeph>INSERT ... VALUES</codeph> statements that insert a single row each
+      time. If you do run <codeph>INSERT ... VALUES</codeph> operations to load data into a staging table as one
+      stage in an ETL pipeline, include multiple row values if possible within each <codeph>VALUES</codeph> clause,
+      and use a separate database to make cleanup easier if the operation does produce many tiny files.
+    </note>
+
+    <p>
+      The following example shows how to insert one row or multiple rows, with expressions of different types,
+      using literal values, expressions, and function return values:
+    </p>
+
+<codeblock>create table val_test_1 (c1 int, c2 float, c3 string, c4 boolean, c5 timestamp);
+insert into val_test_1 values (100, 99.9/10, 'abc', true, now());
+create table val_test_2 (id int, token string);
+insert overwrite val_test_2 values (1, 'a'), (2, 'b'), (-1,'xyzzy');</codeblock>
+
+    <p>
+      These examples show the type of <q>not implemented</q> error that you see when attempting to insert data into
+      a table with a file format that Impala currently does not write to:
+    </p>
+
+<codeblock>DROP TABLE IF EXISTS sequence_table;
+CREATE TABLE sequence_table
+( id INT, col_1 BOOLEAN, col_2 DOUBLE, col_3 TIMESTAMP )
+STORED AS SEQUENCEFILE;
+
+DROP TABLE IF EXISTS rc_table;
+CREATE TABLE rc_table
+( id INT, col_1 BOOLEAN, col_2 DOUBLE, col_3 TIMESTAMP )
+STORED AS RCFILE;
+
+[localhost:21000] &gt; insert into table rc_table select * from default.tab1;
+Remote error
+Backend 0:RC_FILE not implemented.
+
+[localhost:21000] &gt; insert into table sequence_table select * from default.tab1;
+Remote error
+Backend 0:SEQUENCE_FILE not implemented. </codeblock>
+
+    <p>
+      Inserting data into partitioned tables requires slightly different syntax that divides the partitioning
+      columns from the others:
+    </p>
+
+<codeblock>create table t1 (i int) <b>partitioned by (x int, y string)</b>;
+-- Select an INT column from another table.
+-- All inserted rows will have the same x and y values, as specified in the INSERT statement.
+-- This technique of specifying all the partition key values is known as static partitioning.
+insert into t1 <b>partition(x=10, y='a')</b> select c1 from some_other_table;
+-- Select two INT columns from another table.
+-- All inserted rows will have the same y value, as specified in the INSERT statement.
+-- Values from c2 go into t1.x.
+-- Any partitioning columns whose value is not specified are filled in
+-- from the columns specified last in the SELECT list.
+-- This technique of omitting some partition key values is known as dynamic partitioning.
+insert into t1 <b>partition(x, y='b')</b> select c1, c2 from some_other_table;
+-- Select an INT and a STRING column from another table.
+-- All inserted rows will have the same x value, as specified in the INSERT statement.
+-- Values from c3 go into t1.y.
+insert into t1 <b>partition(x=20, y)</b> select c1, c3  from some_other_table;</codeblock>
+
+    <p rev="1.1">
+      The following examples show how you can copy the data in all the columns from one table to another, copy the
+      data from only some columns, or specify the columns in the select list in a different order than they
+      actually appear in the table:
+    </p>
+
+<codeblock>-- Start with 2 identical tables.
+create table t1 (c1 int, c2 int);
+create table t2 like t1;
+
+-- If there is no () part after the destination table name,
+-- all columns must be specified, either as * or by name.
+insert into t2 select * from t1;
+insert into t2 select c1, c2 from t1;
+
+-- With the () notation following the destination table name,
+-- you can omit columns (all values for that column are NULL
+-- in the destination table), and/or reorder the values
+-- selected from the source table. This is the "column permutation" feature.
+insert into t2 (c1) select c1 from t1;
+insert into t2 (c2, c1) select c1, c2 from t1;
+
+-- The column names can be entirely different in the source and destination tables.
+-- You can copy any columns, not just the corresponding ones, from the source table.
+-- But the number and type of selected columns must match the columns mentioned in the () part.
+alter table t2 replace columns (x int, y int);
+insert into t2 (y) select c1 from t1;
+
+-- For partitioned tables, all the partitioning columns must be mentioned in the () column list
+-- or a PARTITION clause; these columns cannot be defaulted to NULL.
+create table pt1 (x int, y int) partitioned by (z int);
+-- The values from c1 are copied into the column x in the new table,
+-- all in the same partition based on a constant value for z.
+-- The values of y in the new table are all NULL.
+insert into pt1 (x) partition (z=5) select c1 from t1;
+-- Again we omit the values for column y so they are all NULL.
+-- The inserted x values can go into different partitions, based on
+-- the different values inserted into the partitioning column z.
+insert into pt1 (x,z) select x, z from t2;
+</codeblock>
+
+    <p>
+      <codeph>SELECT *</codeph> for a partitioned table requires that all partition key columns in the source table
+      be declared as the last columns in the <codeph>CREATE TABLE</codeph> statement. You still include a
+      <codeph>PARTITION BY</codeph> clause listing all the partition key columns. These partition columns are
+      automatically mapped to the last columns from the <codeph>SELECT *</codeph> list.
+    </p>
+
+<codeblock>create table source (x int, y int, year int, month int, day int);
+create table destination (x int, y int) partitioned by (year int, month int, day int);
+...load some data into the unpartitioned source table...
+-- Insert a single partition of data.
+-- The SELECT * means you cannot specify partition (year=2014, month, day).
+insert overwrite destination partition (year, month, day) select * from source where year=2014;
+-- Insert the data for all year/month/day combinations.
+insert overwrite destination partition (year, month, day) select * from source;
+
+-- If one of the partition columns is omitted from the source table,
+-- then you can specify a specific value for that column in the PARTITION clause.
+-- Here the source table holds only data from 2014, and so does not include a year column.
+create table source_2014 (x int, y int, month, day);
+...load some data into the unpartitioned source_2014 table...
+insert overwrite destination partition (year=2014, month, day) select * from source_2014;
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/insert_sort_blurb"/>
+
+    <p>
+      <b>Concurrency considerations:</b> Each <codeph>INSERT</codeph> operation creates new data files with unique
+      names, so you can run multiple <codeph>INSERT INTO</codeph> statements simultaneously without filename
+      conflicts.
+<!--
+If data is inserted into a table by a statement issued to a different
+<cmdname>impalad</cmdname> node,
+issue a <codeph>REFRESH <varname>table_name</varname></codeph>
+statement to make the node you are connected to aware of this new data.
+-->
+      While data is being inserted into an Impala table, the data is staged temporarily in a subdirectory inside
+      the data directory; during this period, you cannot issue queries against that table in Hive. If an
+      <codeph>INSERT</codeph> operation fails, the temporary data file and the subdirectory could be left behind in
+      the data directory. If so, remove the relevant subdirectory and any data files it contains manually, by
+      issuing an <codeph>hdfs dfs -rm -r</codeph> command, specifying the full path of the work subdirectory, whose
+      name ends in <codeph>_dir</codeph>.
+    </p>
+  </conbody>
+
+  <concept id="values">
+
+    <title>VALUES Clause</title>
+
+    <conbody>
+
+      <p>
+        The <codeph>VALUES</codeph> clause is a general-purpose way to specify the columns of one or more rows,
+        typically within an <codeph><xref href="impala_insert.xml#insert">INSERT</xref></codeph> statement.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/insert_values_warning">
+        <p/>
+      </note>
+
+      <p>
+        The following examples illustrate:
+      </p>
+
+      <ul>
+        <li>
+          How to insert a single row using a <codeph>VALUES</codeph> clause.
+        </li>
+
+        <li>
+          How to insert multiple rows using a <codeph>VALUES</codeph> clause.
+        </li>
+
+        <li>
+          How the row or rows from a <codeph>VALUES</codeph> clause can be appended to a table through
+          <codeph>INSERT INTO</codeph>, or replace the contents of the table through <codeph>INSERT
+          OVERWRITE</codeph>.
+        </li>
+
+        <li>
+          How the entries in a <codeph>VALUES</codeph> clause can be literals, function results, or any other kind
+          of expression. See <xref href="impala_literals.xml#literals"/> for the notation to use for literal
+          values, especially <xref href="impala_literals.xml#string_literals"/> for quoting and escaping
+          conventions for strings. See <xref href="impala_operators.xml#operators"/> and
+          <xref href="impala_functions.xml#builtins"/> for other things you can include in expressions with the
+          <codeph>VALUES</codeph> clause.
+        </li>
+      </ul>
+
+<codeblock>[localhost:21000] &gt; describe val_example;
+Query: describe val_example
+Query finished, fetching results ...
++-------+---------+---------+
+| name  | type    | comment |
++-------+---------+---------+
+| id    | int     |         |
+| col_1 | boolean |         |
+| col_2 | double  |         |
++-------+---------+---------+
+
+[localhost:21000] &gt; insert into val_example values (1,true,100.0);
+Inserted 1 rows in 0.30s
+[localhost:21000] &gt; select * from val_example;
++----+-------+-------+
+| id | col_1 | col_2 |
++----+-------+-------+
+| 1  | true  | 100   |
++----+-------+-------+
+
+[localhost:21000] &gt; insert overwrite val_example values (10,false,pow(2,5)), (50,true,10/3);
+Inserted 2 rows in 0.16s
+[localhost:21000] &gt; select * from val_example;
++----+-------+-------------------+
+| id | col_1 | col_2             |
++----+-------+-------------------+
+| 10 | false | 32                |
+| 50 | true  | 3.333333333333333 |
++----+-------+-------------------+</codeblock>
+
+      <p>
+        When used in an <codeph>INSERT</codeph> statement, the Impala <codeph>VALUES</codeph> clause can specify
+        some or all of the columns in the destination table, and the columns can be specified in a different order
+        than they actually appear in the table. To specify a different set or order of columns than in the table,
+        use the syntax:
+      </p>
+
+<codeblock>INSERT INTO <varname>destination</varname>
+  (<varname>col_x</varname>, <varname>col_y</varname>, <varname>col_z</varname>)
+  VALUES
+  (<varname>val_x</varname>, <varname>val_y</varname>, <varname>val_z</varname>);
+</codeblock>
+
+      <p>
+        Any columns in the table that are not listed in the <codeph>INSERT</codeph> statement are set to
+        <codeph>NULL</codeph>.
+      </p>
+
+<!--
+      <p>
+        does not support specifying a subset of the
+        columns in the table or specifying the columns in a different order. Use a
+        <codeph>VALUES</codeph> clause with all the column values in the same order as
+        the table definition, using <codeph>NULL</codeph> values for any columns you
+        want to omit from the <codeph>INSERT</codeph> operation.
+      </p>
+-->
+
+      <p>
+        To use a <codeph>VALUES</codeph> clause like a table in other statements, wrap it in parentheses and use
+        <codeph>AS</codeph> clauses to specify aliases for the entire object and any columns you need to refer to:
+      </p>
+
+<codeblock>[localhost:21000] &gt; select * from (values(4,5,6),(7,8,9)) as t;
++---+---+---+
+| 4 | 5 | 6 |
++---+---+---+
+| 4 | 5 | 6 |
+| 7 | 8 | 9 |
++---+---+---+
+[localhost:21000] &gt; select * from (values(1 as c1, true as c2, 'abc' as c3),(100,false,'xyz')) as t;
++-----+-------+-----+
+| c1  | c2    | c3  |
++-----+-------+-----+
+| 1   | true  | abc |
+| 100 | false | xyz |
++-----+-------+-----+</codeblock>
+
+      <p>
+        For example, you might use a tiny table constructed like this from constant literals or function return
+        values as part of a longer statement involving joins or <codeph>UNION ALL</codeph>.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/hdfs_blurb"/>
+
+      <p>
+        Impala physically writes all inserted files under the ownership of its default user, typically
+        <codeph>impala</codeph>. Therefore, this user must have HDFS write permission in the corresponding table
+        directory.
+      </p>
+
+      <p>
+        The permission requirement is independent of the authorization performed by the Sentry framework. (If the
+        connected user is not authorized to insert into a table, Sentry blocks that operation immediately,
+        regardless of the privileges available to the <codeph>impala</codeph> user.) Files created by Impala are
+        not owned by and do not inherit permissions from the connected user.
+      </p>
+
+      <p>
+        The number of data files produced by an <codeph>INSERT</codeph> statement depends on the size of the
+        cluster, the number of data blocks that are processed, the partition key columns in a partitioned table,
+        and the mechanism Impala uses for dividing the work in parallel. Do not assume that an
+        <codeph>INSERT</codeph> statement will produce some particular number of output files. In case of
+        performance issues with data written by Impala, check that the output files do not suffer from issues such
+        as many tiny files or many tiny partitions. (In the Hadoop context, even files or partitions of a few tens
+        of megabytes are considered <q>tiny</q>.)
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/insert_hidden_work_directory"/>
+
+      <p conref="../shared/impala_common.xml#common/hbase_blurb"/>
+
+      <p>
+        You can use the <codeph>INSERT</codeph> statement with HBase tables as follows:
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            You can insert a single row or a small set of rows into an HBase table with the <codeph>INSERT ...
+            VALUES</codeph> syntax. This is a good use case for HBase tables with Impala, because HBase tables are
+            not subject to the same kind of fragmentation from many small insert operations as HDFS tables are.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            You can insert any number of rows at once into an HBase table using the <codeph>INSERT ...
+            SELECT</codeph> syntax.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            If more than one inserted row has the same value for the HBase key column, only the last inserted row
+            with that value is visible to Impala queries. You can take advantage of this fact with <codeph>INSERT
+            ... VALUES</codeph> statements to effectively update rows one at a time, by inserting new rows with the
+            same key values as existing rows. Be aware that after an <codeph>INSERT ... SELECT</codeph> operation
+            copying from an HDFS table, the HBase table might contain fewer rows than were inserted, if the key
+            column in the source table contained duplicate values.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            You cannot <codeph>INSERT OVERWRITE</codeph> into an HBase table. New rows are always appended.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            When you create an Impala or Hive table that maps to an HBase table, the column order you specify with
+            the <codeph>INSERT</codeph> statement might be different than the order you declare with the
+            <codeph>CREATE TABLE</codeph> statement. Behind the scenes, HBase arranges the columns based on how
+            they are divided into column families. This might cause a mismatch during insert operations, especially
+            if you use the syntax <codeph>INSERT INTO <varname>hbase_table</varname> SELECT * FROM
+            <varname>hdfs_table</varname></codeph>. Before inserting data, verify the column order by issuing a
+            <codeph>DESCRIBE</codeph> statement for the table, and adjust the order of the select list in the
+            <codeph>INSERT</codeph> statement.
+          </p>
+        </li>
+      </ul>
+
+      <p>
+        See <xref href="impala_hbase.xml#impala_hbase"/> for more details about using Impala with HBase.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/s3_blurb"/>
+      <p conref="../shared/impala_common.xml#common/s3_dml"/>
+      <p conref="../shared/impala_common.xml#common/s3_dml_performance"/>
+      <p>See <xref href="../topics/impala_s3.xml#s3"/> for details about reading and writing S3 data with Impala.</p>
+
+      <p conref="../shared/impala_common.xml#common/security_blurb"/>
+      <p conref="../shared/impala_common.xml#common/redaction_yes"/>
+
+      <p conref="../shared/impala_common.xml#common/cancel_blurb_yes"/>
+
+      <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+      <p rev="CDH-19187">
+        The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+        typically the <codeph>impala</codeph> user, must have read
+        permission for the files in the source directory of an <codeph>INSERT ... SELECT</codeph>
+        operation, and write permission for all affected directories in the destination table.
+        (An <codeph>INSERT</codeph> operation could write files to multiple different HDFS directories
+        if the destination table is partitioned.)
+        This user must also have write permission to create a temporary work directory
+        in the top-level HDFS directory of the destination table.
+        An <codeph>INSERT OVERWRITE</codeph> operation does not require write permission on
+        the original data files in the table, only on the table directories themselves.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/char_varchar_cast_from_string"/>
+
+      <p conref="../shared/impala_common.xml#common/related_options"/>
+
+      <p rev="1.3.1" conref="../shared/impala_common.xml#common/insert_inherit_permissions"/>
+    </conbody>
+  </concept>
+
+<!-- Values clause -->
+</concept>
+<!-- INSERT statement -->

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_install.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_install.xml b/docs/topics/impala_install.xml
new file mode 100644
index 0000000..ab68314
--- /dev/null
+++ b/docs/topics/impala_install.xml
@@ -0,0 +1,106 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="install">
+
+  <title><ph audience="standalone">Installing Impala</ph><ph audience="integrated">Impala Installation</ph></title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Installing"/>
+      <data name="Category" value="Administrators"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">installation</indexterm>
+      <indexterm audience="Cloudera">pseudo-distributed cluster</indexterm>
+      <indexterm audience="Cloudera">cluster</indexterm>
+      <indexterm audience="Cloudera">DataNodes</indexterm>
+      <indexterm audience="Cloudera">NameNode</indexterm>
+      <indexterm audience="Cloudera">Cloudera Manager</indexterm>
+      <indexterm audience="Cloudera">impalad</indexterm>
+      <indexterm audience="Cloudera">impala-shell</indexterm>
+      <indexterm audience="Cloudera">statestored</indexterm>
+      Impala is an open-source add-on to the Cloudera Enterprise Core that returns rapid responses to
+      queries.
+    </p>
+
+    <note>
+      <p>
+        Under CDH 5, Impala is included as part of the CDH installation and no separate steps are needed.
+        <ph audience="standalone">Therefore, the instruction steps in this section apply to CDH 4 only.</ph>
+      </p>
+    </note>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="install_details">
+
+    <title>What is Included in an Impala Installation</title>
+
+    <conbody>
+
+      <p>
+        Impala is made up of a set of components that can be installed on multiple nodes throughout your cluster.
+        The key installation step for performance is to install the <cmdname>impalad</cmdname> daemon (which does
+        most of the query processing work) on <i>all</i> DataNodes in the cluster.
+      </p>
+
+      <p>
+        The Impala package installs these binaries:
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            <cmdname>impalad</cmdname> - The Impala daemon. Plans and executes queries against HDFS, HBase, <ph rev="2.2.0">and Amazon S3 data</ph>.
+            <xref href="impala_processes.xml#processes">Run one impalad process</xref> on each node in the cluster
+            that has a DataNode.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            <cmdname>statestored</cmdname> - Name service that tracks location and status of all
+            <codeph>impalad</codeph> instances in the cluster. <xref href="impala_processes.xml#processes">Run one
+            instance of this daemon</xref> on a node in your cluster. Most production deployments run this daemon
+            on the namenode.
+          </p>
+        </li>
+
+        <li rev="1.2">
+          <p>
+            <cmdname>catalogd</cmdname> - Metadata coordination service that broadcasts changes from Impala DDL and
+            DML statements to all affected Impala nodes, so that new tables, newly loaded data, and so on are
+            immediately visible to queries submitted through any Impala node.
+<!-- Consider removing this when 1.2 gets far in the past. -->
+            (Prior to Impala 1.2, you had to run the <codeph>REFRESH</codeph> or <codeph>INVALIDATE
+            METADATA</codeph> statement on each node to synchronize changed metadata. Now those statements are only
+            required if you perform the DDL or DML through an external mechanism such as Hive <ph rev="2.2.0">or by uploading
+            data to the Amazon S3 filesystem</ph>.)
+            <xref href="impala_processes.xml#processes">Run one instance of this daemon</xref> on a node in your cluster,
+            preferably on the same host as the <codeph>statestored</codeph> daemon.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            <cmdname>impala-shell</cmdname> - <xref href="impala_impala_shell.xml#impala_shell">Command-line
+            interface</xref> for issuing queries to the Impala daemon. You install this on one or more hosts
+            anywhere on your network, not necessarily DataNodes or even within the same cluster as Impala. It can
+            connect remotely to any instance of the Impala daemon.
+          </p>
+        </li>
+      </ul>
+
+      <p>
+        Before doing the installation, ensure that you have all necessary prerequisites. See
+        <xref href="impala_prereqs.xml#prereqs"/> for details.
+      </p>
+    </conbody>
+  </concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_int.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_int.xml b/docs/topics/impala_int.xml
new file mode 100644
index 0000000..aeead5b
--- /dev/null
+++ b/docs/topics/impala_int.xml
@@ -0,0 +1,95 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="int">
+
+  <title>INT Data Type</title>
+  <titlealts audience="PDF"><navtitle>INT</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      A 4-byte integer data type used in <codeph>CREATE TABLE</codeph> and <codeph>ALTER TABLE</codeph> statements.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p>
+      In the column definition of a <codeph>CREATE TABLE</codeph> statement:
+    </p>
+
+<codeblock><varname>column_name</varname> INT</codeblock>
+
+    <p>
+      <b>Range:</b> -2147483648 .. 2147483647. There is no <codeph>UNSIGNED</codeph> subtype.
+    </p>
+
+    <p>
+      <b>Conversions:</b> Impala automatically converts to a larger integer type (<codeph>BIGINT</codeph>) or a
+      floating-point type (<codeph>FLOAT</codeph> or <codeph>DOUBLE</codeph>) automatically. Use
+      <codeph>CAST()</codeph> to convert to <codeph>TINYINT</codeph>, <codeph>SMALLINT</codeph>,
+      <codeph>STRING</codeph>, or <codeph>TIMESTAMP</codeph>.
+      <ph conref="../shared/impala_common.xml#common/cast_int_to_timestamp"/>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      The data type <codeph>INTEGER</codeph> is an alias for <codeph>INT</codeph>.
+    </p>
+
+    <p>
+      For a convenient and automated way to check the bounds of the <codeph>INT</codeph> type, call the functions
+      <codeph>MIN_INT()</codeph> and <codeph>MAX_INT()</codeph>.
+    </p>
+
+    <p>
+      If an integer value is too large to be represented as a <codeph>INT</codeph>, use a <codeph>BIGINT</codeph>
+      instead.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/null_bad_numeric_cast"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>CREATE TABLE t1 (x INT);
+SELECT CAST(1000 AS INT);
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/partitioning_good"/>
+
+    <p conref="../shared/impala_common.xml#common/hbase_ok"/>
+
+    <p conref="../shared/impala_common.xml#common/parquet_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/text_bulky"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/compatibility_blurb"/> -->
+
+    <p conref="../shared/impala_common.xml#common/internals_4_bytes"/>
+
+    <p conref="../shared/impala_common.xml#common/added_forever"/>
+
+    <p conref="../shared/impala_common.xml#common/column_stats_constant"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/restrictions_blurb"/> -->
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_literals.xml#numeric_literals"/>, <xref href="impala_tinyint.xml#tinyint"/>,
+      <xref href="impala_smallint.xml#smallint"/>, <xref href="impala_int.xml#int"/>,
+      <xref href="impala_bigint.xml#bigint"/>, <xref href="impala_decimal.xml#decimal"/>,
+      <xref href="impala_math_functions.xml#math_functions"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_intro.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_intro.xml b/docs/topics/impala_intro.xml
new file mode 100644
index 0000000..c599bc5
--- /dev/null
+++ b/docs/topics/impala_intro.xml
@@ -0,0 +1,81 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="intro">
+
+  <title id="impala"><ph audience="standalone">Introducing Apache Impala (incubating)</ph><ph audience="integrated">Apache Impala (incubating) Overview</ph></title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Getting Started"/>
+      <data name="Category" value="Concepts"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+    </metadata>
+  </prolog>
+
+  <conbody id="intro_body">
+
+      <p>
+        Impala provides fast, interactive SQL queries directly on your Apache Hadoop data stored in HDFS,
+        HBase, <ph rev="2.2.0">or the Amazon Simple Storage Service (S3)</ph>.
+        In addition to using the same unified storage platform,
+        Impala also uses the same metadata, SQL syntax (Hive SQL), ODBC driver, and user interface
+        (Impala query UI in Hue) as Apache Hive. This
+        provides a familiar and unified platform for real-time or batch-oriented queries.
+      </p>
+
+      <p>
+        Impala is an addition to tools available for querying big data. Impala does not replace the batch
+        processing frameworks built on MapReduce such as Hive. Hive and other frameworks built on MapReduce are
+        best suited for long running batch jobs, such as those involving batch processing of Extract, Transform,
+        and Load (ETL) type jobs.
+      </p>
+
+      <note>
+        Impala was accepted into the Apache incubator on December 2, 2015.
+        In places where the documentation formerly referred to <q>Cloudera Impala</q>,
+        now the official name is <q>Apache Impala (incubating)</q>.
+      </note>
+
+  </conbody>
+
+  <concept id="benefits">
+
+    <title>Impala Benefits</title>
+
+    <conbody>
+
+      <p conref="../shared/impala_common.xml#common/impala_benefits"/>
+
+    </conbody>
+  </concept>
+
+  <concept id="impala_cdh">
+
+    <title>How Impala Works with CDH</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Concepts"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p conref="../shared/impala_common.xml#common/impala_overview_diagram"/>
+
+      <p conref="../shared/impala_common.xml#common/component_list"/>
+
+      <p conref="../shared/impala_common.xml#common/query_overview"/>
+    </conbody>
+  </concept>
+
+  <concept id="features">
+
+    <title>Primary Impala Features</title>
+
+    <conbody>
+
+      <p conref="../shared/impala_common.xml#common/feature_list"/>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_invalidate_metadata.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_invalidate_metadata.xml b/docs/topics/impala_invalidate_metadata.xml
new file mode 100644
index 0000000..c41a996
--- /dev/null
+++ b/docs/topics/impala_invalidate_metadata.xml
@@ -0,0 +1,233 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.1" id="invalidate_metadata">
+
+  <title>INVALIDATE METADATA Statement</title>
+  <titlealts audience="PDF"><navtitle>INVALIDATE METADATA</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="ETL"/>
+      <data name="Category" value="Ingest"/>
+      <data name="Category" value="Metastore"/>
+      <data name="Category" value="Schemas"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">INVALIDATE METADATA statement</indexterm>
+      Marks the metadata for one or all tables as stale. Required after a table is created through the Hive shell,
+      before the table is available for Impala queries. The next time the current Impala node performs a query
+      against a table whose metadata is invalidated, Impala reloads the associated metadata before the query
+      proceeds. This is a relatively expensive operation compared to the incremental metadata update done by the
+      <codeph>REFRESH</codeph> statement, so in the common scenario of adding new data files to an existing table,
+      prefer <codeph>REFRESH</codeph> rather than <codeph>INVALIDATE METADATA</codeph>. If you are not familiar
+      with the way Impala uses metadata and how it shares the same metastore database as Hive, see
+      <xref href="impala_hadoop.xml#intro_metastore"/> for background information.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>INVALIDATE METADATA [[<varname>db_name</varname>.]<varname>table_name</varname>]</codeblock>
+
+    <p>
+      By default, the cached metadata for all tables is flushed. If you specify a table name, only the metadata for
+      that one table is flushed. Even for a single table, <codeph>INVALIDATE METADATA</codeph> is more expensive
+      than <codeph>REFRESH</codeph>, so prefer <codeph>REFRESH</codeph> in the common case where you add new data
+      files for an existing table.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/internals_blurb"/>
+
+    <p>
+      To accurately respond to queries, Impala must have current metadata about those databases and tables that
+      clients query directly. Therefore, if some other entity modifies information used by Impala in the metastore
+      that Impala and Hive share, the information cached by Impala must be updated. However, this does not mean
+      that all metadata updates require an Impala update.
+    </p>
+
+    <note>
+      <p conref="../shared/impala_common.xml#common/catalog_server_124"/>
+      <p rev="1.2">
+        In Impala 1.2 and higher, a dedicated daemon (<cmdname>catalogd</cmdname>) broadcasts DDL changes made
+        through Impala to all Impala nodes. Formerly, after you created a database or table while connected to one
+        Impala node, you needed to issue an <codeph>INVALIDATE METADATA</codeph> statement on another Impala node
+        before accessing the new database or table from the other node. Now, newly created or altered objects are
+        picked up automatically by all Impala nodes. You must still use the <codeph>INVALIDATE METADATA</codeph>
+        technique after creating or altering objects through Hive. See
+        <xref href="impala_components.xml#intro_catalogd"/> for more information on the catalog service.
+      </p>
+      <p>
+        The <codeph>INVALIDATE METADATA</codeph> statement is new in Impala 1.1 and higher, and takes over some of
+        the use cases of the Impala 1.0 <codeph>REFRESH</codeph> statement. Because <codeph>REFRESH</codeph> now
+        requires a table name parameter, to flush the metadata for all tables at once, use the <codeph>INVALIDATE
+        METADATA</codeph> statement.
+      </p>
+      <p conref="../shared/impala_common.xml#common/invalidate_then_refresh"/>
+    </note>
+
+    <p conref="../shared/impala_common.xml#common/refresh_vs_invalidate"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      A metadata update for an <codeph>impalad</codeph> instance <b>is</b> required if:
+    </p>
+
+    <ul>
+      <li>
+        A metadata change occurs.
+      </li>
+
+      <li>
+        <b>and</b> the change is made from another <codeph>impalad</codeph> instance in your cluster, or through
+        Hive.
+      </li>
+
+      <li>
+        <b>and</b> the change is made to a metastore database to which clients such as the Impala shell or ODBC directly
+        connect.
+      </li>
+    </ul>
+
+    <p>
+      A metadata update for an Impala node is <b>not</b> required when you issue queries from the same Impala node
+      where you ran <codeph>ALTER TABLE</codeph>, <codeph>INSERT</codeph>, or other table-modifying statement.
+    </p>
+
+    <p>
+      Database and table metadata is typically modified by:
+    </p>
+
+    <ul>
+      <li>
+        Hive - via <codeph>ALTER</codeph>, <codeph>CREATE</codeph>, <codeph>DROP</codeph> or
+        <codeph>INSERT</codeph> operations.
+      </li>
+
+      <li>
+        Impalad - via <codeph>CREATE TABLE</codeph>, <codeph>ALTER TABLE</codeph>, and <codeph>INSERT</codeph>
+        operations.
+      </li>
+    </ul>
+
+    <p>
+      <codeph>INVALIDATE METADATA</codeph> causes the metadata for that table to be marked as stale, and reloaded
+      the next time the table is referenced. For a huge table, that process could take a noticeable amount of time;
+      thus you might prefer to use <codeph>REFRESH</codeph> where practical, to avoid an unpredictable delay later,
+      for example if the next reference to the table is during a benchmark test.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      The following example shows how you might use the <codeph>INVALIDATE METADATA</codeph> statement after
+      creating new tables (such as SequenceFile or HBase tables) through the Hive shell. Before the
+      <codeph>INVALIDATE METADATA</codeph> statement was issued, Impala would give a <q>table not found</q> error
+      if you tried to refer to those table names. The <codeph>DESCRIBE</codeph> statements cause the latest
+      metadata to be immediately loaded for the tables, avoiding a delay the next time those tables are queried.
+    </p>
+
+<codeblock>[impalad-host:21000] &gt; invalidate metadata;
+[impalad-host:21000] &gt; describe t1;
+...
+[impalad-host:21000] &gt; describe t2;
+... </codeblock>
+
+    <p>
+      For more examples of using <codeph>REFRESH</codeph> and <codeph>INVALIDATE METADATA</codeph> with a
+      combination of Impala and Hive operations, see <xref href="impala_tutorial.xml#tutorial_impala_hive"/>.
+    </p>
+
+    <p>
+      If you need to ensure that the metadata is up-to-date when you start an <cmdname>impala-shell</cmdname>
+      session, run <cmdname>impala-shell</cmdname> with the <codeph>-r</codeph> or
+      <codeph>--refresh_after_connect</codeph> command-line option. Because this operation adds a delay to the next
+      query against each table, potentially expensive for large tables with many partitions, try to avoid using
+      this option for day-to-day operations in a production environment.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+    <p rev="CDH-19187">
+      The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+      typically the <codeph>impala</codeph> user, must have execute
+      permissions for all the relevant directories holding table data.
+      (A table could have data spread across multiple directories,
+      or in unexpected paths, if it uses partitioning or
+      specifies a <codeph>LOCATION</codeph> attribute for
+      individual partitions or the entire table.)
+      Issues with permissions might not cause an immediate error for this statement,
+      but subsequent statements such as <codeph>SELECT</codeph>
+      or <codeph>SHOW TABLE STATS</codeph> could fail.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/hdfs_blurb"/>
+
+    <p>
+      By default, the <codeph>INVALIDATE METADATA</codeph> command checks HDFS permissions of the underlying data
+      files and directories, caching this information so that a statement can be cancelled immediately if for
+      example the <codeph>impala</codeph> user does not have permission to write to the data directory for the
+      table. (This checking does not apply if you have set the <cmdname>catalogd</cmdname> configuration option
+      <codeph>--load_catalog_in_background=false</codeph>.) Impala reports any lack of write permissions as an
+      <codeph>INFO</codeph> message in the log file, in case that represents an oversight. If you change HDFS
+      permissions to make data readable or writeable by the Impala user, issue another <codeph>INVALIDATE
+      METADATA</codeph> to make Impala aware of the change.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p rev="1.2.4">
+      This example illustrates creating a new database and new table in Hive, then doing an <codeph>INVALIDATE
+      METADATA</codeph> statement in Impala using the fully qualified table name, after which both the new table
+      and the new database are visible to Impala. The ability to specify <codeph>INVALIDATE METADATA
+      <varname>table_name</varname></codeph> for a table created in Hive is a new capability in Impala 1.2.4. In
+      earlier releases, that statement would have returned an error indicating an unknown table, requiring you to
+      do <codeph>INVALIDATE METADATA</codeph> with no table name, a more expensive operation that reloaded metadata
+      for all tables and databases.
+    </p>
+
+<codeblock rev="1.2.4">$ hive
+hive&gt; create database new_db_from_hive;
+OK
+Time taken: 4.118 seconds
+hive&gt; create table new_db_from_hive.new_table_from_hive (x int);
+OK
+Time taken: 0.618 seconds
+hive&gt; quit;
+$ impala-shell
+[localhost:21000] &gt; show databases like 'new*';
+[localhost:21000] &gt; refresh new_db_from_hive.new_table_from_hive;
+ERROR: AnalysisException: Database does not exist: new_db_from_hive
+[localhost:21000] &gt; invalidate metadata new_db_from_hive.new_table_from_hive;
+[localhost:21000] &gt; show databases like 'new*';
++--------------------+
+| name               |
++--------------------+
+| new_db_from_hive   |
++--------------------+
+[localhost:21000] &gt; show tables in new_db_from_hive;
++---------------------+
+| name                |
++---------------------+
+| new_table_from_hive |
++---------------------+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/s3_blurb"/>
+    <p conref="../shared/impala_common.xml#common/s3_metadata"/>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_hadoop.xml#intro_metastore"/>,
+      <xref href="impala_refresh.xml#refresh"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_isilon.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_isilon.xml b/docs/topics/impala_isilon.xml
new file mode 100644
index 0000000..f631268
--- /dev/null
+++ b/docs/topics/impala_isilon.xml
@@ -0,0 +1,114 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="5.4.3" id="impala_isilon">
+
+  <title>Using Impala with Isilon Storage</title>
+  <titlealts audience="PDF"><navtitle>Isilon Storage</navtitle></titlealts>
+
+  <prolog>
+    <metadata>
+      <data name="Category" value="CDH"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Isilon"/>
+      <data name="Category" value="Disk Storage"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">Isilon</indexterm>
+      You can use Impala to query data files that reside on EMC Isilon storage devices, rather than in HDFS.
+      This capability allows convenient query access to a storage system where you might already be
+      managing large volumes of data. The combination of the Impala query engine and Isilon storage is
+      certified on CDH 5.4.4 or higher.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/isilon_block_size_caveat"/>
+
+    <p>
+      The typical use case for Impala and Isilon together is to use Isilon for the
+      default filesystem, replacing HDFS entirely. In this configuration,
+      when you create a database, table, or partition, the data always resides on
+      Isilon storage and you do not need to specify any special <codeph>LOCATION</codeph>
+      attribute. If you do specify a <codeph>LOCATION</codeph> attribute, its value refers
+      to a path within the Isilon filesystem.
+      For example:
+    </p>
+<codeblock>-- If the default filesystem is Isilon, all Impala data resides there
+-- and all Impala databases and tables are located there.
+CREATE TABLE t1 (x INT, s STRING);
+
+-- You can specify LOCATION for database, table, or partition,
+-- using values from the Isilon filesystem.
+CREATE DATABASE d1 LOCATION '/some/path/on/isilon/server/d1.db';
+CREATE TABLE d1.t2 (a TINYINT, b BOOLEAN);
+</codeblock>
+
+    <p>
+      Impala can write to, delete, and rename data files and database, table,
+      and partition directories on Isilon storage. Therefore, Impala statements such
+      as
+      <codeph>CREATE TABLE</codeph>, <codeph>DROP TABLE</codeph>,
+      <codeph>CREATE DATABASE</codeph>, <codeph>DROP DATABASE</codeph>,
+      <codeph>ALTER TABLE</codeph>,
+      and
+      <codeph>INSERT</codeph> work the same with Isilon storage as with HDFS.
+    </p>
+
+    <p>
+      When the Impala spill-to-disk feature is activated by a query that approaches
+      the memory limit, Impala writes all the temporary data to a local (not Isilon)
+      storage device. Because the I/O bandwidth for the temporary data depends on
+      the number of local disks, and clusters using Isilon storage might not have
+      as many local disks attached, pay special attention on Isilon-enabled clusters
+      to any queries that use the spill-to-disk feature. Where practical, tune the
+      queries or allocate extra memory for Impala to avoid spilling.
+      Although you can specify an Isilon storage device as the destination for
+      the temporary data for the spill-to-disk feature, that configuration is
+      not recommended due to the need to transfer the data both ways using remote I/O.
+    </p>
+
+    <p>
+      When tuning Impala queries on HDFS, you typically try to avoid any remote reads.
+      When the data resides on Isilon storage, all the I/O consists of remote reads.
+      Do not be alarmed when you see non-zero numbers for remote read measurements
+      in query profile output. The benefit of the Impala and Isilon integration is
+      primarily convenience of not having to move or copy large volumes of data to HDFS,
+      rather than raw query performance. You can increase the performance of Impala
+      I/O for Isilon systems by increasing the value for the
+      <codeph>num_remote_hdfs_io_threads</codeph> configuration parameter,
+      in the Cloudera Manager user interface for clusters using Cloudera Manager,
+      or through the <codeph>--num_remote_hdfs_io_threads</codeph> startup option
+      for the <cmdname>impalad</cmdname> daemon on clusters not using Cloudera Manager.
+    </p>
+
+    <p>
+<!--
+      For information about tasks performed on
+      Isilon OneFS, see the information hub for Cloudera on the EMC Community Network:
+      <xref href="https://community.emc.com/docs/DOC-39522" format="html" scope="external">https://community.emc.com/docs/DOC-39522</xref>.
+-->
+      <!-- This is a little bit of a circular loop when this topic is conrefed into the main Isilon page,
+           consider if there's a way to conditionalize it out in that case. -->
+      For information about managing Isilon storage devices through Cloudera Manager, see
+      <xref audience="integrated" href="cm_mc_isilon_service.xml"/><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_isilon_service.html" scope="external" format="html"/>.
+    </p>
+
+    <!-- <p outputclass="toc inpage"/> -->
+  </conbody>
+<concept id="isilon_cm_configs">
+<title>Required Configurations</title>
+<conbody>
+<p>Specify the following configurations in Cloudera Manager on the <menucascade><uicontrol>Clusters</uicontrol><uicontrol><varname>Isilon Service</varname></uicontrol><uicontrol>Configuration</uicontrol></menucascade> tab:<ul id="ul_vpx_bw5_vv">
+<li>In <uicontrol>HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml</uicontrol> <codeph>hdfs-site.xml</codeph> and the <uicontrol>Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml</uicontrol> properties for the Isilon service, set the value of the <codeph>dfs.client.file-block-storage-locations.timeout.millis</codeph> property to <codeph>10000</codeph>.</li>
+<li>In the Isilon <uicontrol>Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml</uicontrol> property for the Isilon service, set the value of the <codeph>hadoop.security.token.service.use_ip</codeph> property to <codeph>FALSE</codeph>. </li>
+<li>If you see errors that reference the <codeph>.Trash</codeph> directory, make sure that the <uicontrol>Use Trash</uicontrol> property is selected.</li>
+</ul></p>
+
+</conbody>
+</concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_jdbc.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_jdbc.xml b/docs/topics/impala_jdbc.xml
new file mode 100644
index 0000000..d667170
--- /dev/null
+++ b/docs/topics/impala_jdbc.xml
@@ -0,0 +1,387 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="impala_jdbc">
+
+  <title id="jdbc">Configuring Impala to Work with JDBC</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="JDBC"/>
+      <data name="Category" value="Java"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Configuring"/>
+      <data name="Category" value="Starting and Stopping"/>
+      <data name="Category" value="Developers"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">JDBC</indexterm>
+      Impala supports the standard JDBC interface, allowing access from commercial Business Intelligence tools and
+      custom software written in Java or other programming languages. The JDBC driver allows you to access Impala
+      from a Java program that you write, or a Business Intelligence or similar tool that uses JDBC to communicate
+      with various database products.
+    </p>
+
+    <p>
+      Setting up a JDBC connection to Impala involves the following steps:
+    </p>
+
+    <ul>
+      <li>
+        Verifying the communication port where the Impala daemons in your cluster are listening for incoming JDBC
+        requests.
+      </li>
+
+      <li>
+        Installing the JDBC driver on every system that runs the JDBC-enabled application.
+      </li>
+
+      <li>
+        Specifying a connection string for the JDBC application to access one of the servers running the
+        <cmdname>impalad</cmdname> daemon, with the appropriate security settings.
+      </li>
+    </ul>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="jdbc_port">
+
+    <title>Configuring the JDBC Port</title>
+
+    <conbody>
+
+      <p>
+        The default port used by JDBC 2.0 and later (as well as ODBC 2.x) is 21050. Impala server accepts JDBC
+        connections through this same port 21050 by default. Make sure this port is available for communication
+        with other hosts on your network, for example, that it is not blocked by firewall software. If your JDBC
+        client software connects to a different port, specify that alternative port number with the
+        <codeph>--hs2_port</codeph> option when starting <codeph>impalad</codeph>. See
+        <xref href="impala_processes.xml#processes"/> for details about Impala startup options. See
+        <xref href="impala_ports.xml#ports"/> for information about all ports used for communication between Impala
+        and clients or between Impala components.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="jdbc_driver_choice">
+
+    <title>Choosing the JDBC Driver</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Planning"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        In Impala 2.0 and later, you have the choice between the Cloudera JDBC Connector and the Hive 0.13 JDBC driver.
+        <ph rev="upstream">Cloudera</ph> recommends using the Cloudera JDBC Connector where practical.
+      </p>
+
+      <p>
+        If you are already using JDBC applications with an earlier Impala release, you must update your JDBC driver
+        to one of these choices, because the Hive 0.12 driver that was formerly the only choice is not compatible
+        with Impala 2.0 and later.
+      </p>
+
+      <p>
+        Both the Cloudera JDBC 2.5 Connector and the Hive JDBC driver provide a substantial speed increase for JDBC
+        applications with Impala 2.0 and higher, for queries that return large result sets.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/jdbc_odbc_complex_types"/>
+      <p conref="../shared/impala_common.xml#common/jdbc_odbc_complex_types_views"/>
+
+    </conbody>
+  </concept>
+
+  <concept id="jdbc_setup">
+
+    <title>Enabling Impala JDBC Support on Client Systems</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Installing"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <section id="install_jdbc_connector">
+        <title>Using the Cloudera JDBC Connector (recommended)</title>
+
+        <p>
+          You download and install the Cloudera JDBC 2.5 connector on any Linux, Windows, or Mac system where you
+          intend to run JDBC-enabled applications. From the
+          <xref href="http://go.cloudera.com/odbc-driver-hive-impala.html" scope="external" format="html">Cloudera
+          Connectors download page</xref>, you choose the appropriate protocol (JDBC or ODBC) and target product
+          (Impala or Hive). The ease of downloading and installing on non-CDH systems makes this connector a
+          convenient choice for organizations with heterogeneous environments.
+        </p>
+
+      </section>
+
+      <section id="install_hive_driver">
+        <title>Using the Hive JDBC Driver</title>
+        <p>
+          You install the Hive JDBC driver (<codeph>hive-jdbc</codeph> package) through the Linux package manager, on
+          hosts within the CDH cluster. The driver consists of several Java JAR files. The same driver can be used by Impala and Hive.
+        </p>
+
+        <p>
+          To get the JAR files, install the Hive JDBC driver on each CDH-enabled host in the cluster that will run
+          JDBC applications. Follow the instructions for
+          <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_hive_jdbc_install.html" scope="external" format="html">CDH 5</xref>.
+        </p>
+
+        <note>
+          The latest JDBC driver, corresponding to Hive 0.13, provides substantial performance improvements for
+          Impala queries that return large result sets. Impala 2.0 and later are compatible with the Hive 0.13
+          driver. If you already have an older JDBC driver installed, and are running Impala 2.0 or higher, consider
+          upgrading to the latest Hive JDBC driver for best performance with JDBC applications.
+        </note>
+
+        <p>
+          If you are using JDBC-enabled applications on hosts outside the CDH cluster, you cannot use the CDH install
+          procedure on the non-CDH hosts. Install the JDBC driver on at least one CDH host using the preceding
+          procedure. Then download the JAR files to each client machine that will use JDBC with Impala:
+        </p>
+
+  <codeblock>commons-logging-X.X.X.jar
+  hadoop-common.jar
+  hive-common-X.XX.X-cdhX.X.X.jar
+  hive-jdbc-X.XX.X-cdhX.X.X.jar
+  hive-metastore-X.XX.X-cdhX.X.X.jar
+  hive-service-X.XX.X-cdhX.X.X.jar
+  httpclient-X.X.X.jar
+  httpcore-X.X.X.jar
+  libfb303-X.X.X.jar
+  libthrift-X.X.X.jar
+  log4j-X.X.XX.jar
+  slf4j-api-X.X.X.jar
+  slf4j-logXjXX-X.X.X.jar
+  </codeblock>
+
+        <p>
+          <b>To enable JDBC support for Impala on the system where you run the JDBC application:</b>
+        </p>
+
+        <ol>
+          <li>
+            Download the JAR files listed above to each client machine.
+  <!--
+            Download the
+            <xref href="https://downloads.cloudera.com/impala-jdbc/impala-jdbc-0.5-2.zip" scope="external" format="zip">Impala
+            JDBC zip file</xref> to the client machine that you will use to connect to Impala servers.
+  -->
+            <note>
+              For Maven users, see
+              <xref href="https://github.com/onefoursix/Cloudera-Impala-JDBC-Example" scope="external" format="html">this
+              sample github page</xref> for an example of the dependencies you could add to a <codeph>pom</codeph>
+              file instead of downloading the individual JARs.
+            </note>
+          </li>
+
+          <li>
+            Store the JAR files in a location of your choosing, ideally a directory already referenced in your
+            <codeph>CLASSPATH</codeph> setting. For example:
+            <ul>
+              <li>
+                On Linux, you might use a location such as
+                <codeph>/</codeph><codeph>opt</codeph><codeph>/jars/</codeph>.
+              </li>
+
+              <li>
+                On Windows, you might use a subdirectory underneath <filepath>C:\Program Files</filepath>.
+              </li>
+            </ul>
+          </li>
+
+          <li>
+            To successfully load the Impala JDBC driver, client programs must be able to locate the associated JAR
+            files. This often means setting the <codeph>CLASSPATH</codeph> for the client process to include the
+            JARs. Consult the documentation for your JDBC client for more details on how to install new JDBC drivers,
+            but some examples of how to set <codeph>CLASSPATH</codeph> variables include:
+            <ul>
+              <li>
+                On Linux, if you extracted the JARs to <codeph>/opt/jars/</codeph>, you might issue the following
+                command to prepend the JAR files path to an existing classpath:
+  <codeblock>export CLASSPATH=/opt/jars/*.jar:$CLASSPATH</codeblock>
+              </li>
+
+              <li>
+                On Windows, use the <b>System Properties</b> control panel item to modify the <b>Environment
+                Variables</b> for your system. Modify the environment variables to include the path to which you
+                extracted the files.
+                <note>
+                  If the existing <codeph>CLASSPATH</codeph> on your client machine refers to some older version of
+                  the Hive JARs, ensure that the new JARs are the first ones listed. Either put the new JAR files
+                  earlier in the listings, or delete the other references to Hive JAR files.
+                </note>
+              </li>
+            </ul>
+          </li>
+        </ol>
+      </section>
+
+    </conbody>
+  </concept>
+
+  <concept id="jdbc_connect">
+
+    <title>Establishing JDBC Connections</title>
+
+    <conbody>
+
+      <p>
+        The JDBC driver class depends on which driver you select.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/proxy_jdbc_caveat"/>
+
+      <section id="class_jdbc_connector">
+
+        <title>Using the Cloudera JDBC Connector (recommended)</title>
+
+        <p>
+          Depending on the level of the JDBC API your application is targeting, you can use
+          the following fully-qualified class names (FQCNs):
+        </p>
+
+        <ul>
+          <li><codeph>com.cloudera.impala.jdbc41.Driver</codeph></li>
+          <li><codeph>com.cloudera.impala.jdbc41.DataSource</codeph></li>
+        </ul>
+
+        <ul>
+          <li><codeph>com.cloudera.impala.jdbc4.Driver</codeph></li>
+          <li><codeph>com.cloudera.impala.jdbc4.DataSource</codeph></li>
+        </ul>
+
+        <ul>
+          <li><codeph>com.cloudera.impala.jdbc3.Driver</codeph></li>
+          <li><codeph>com.cloudera.impala.jdbc3.DataSource</codeph></li>
+        </ul>
+
+        <p>
+          The connection string has the following format:
+        </p>
+
+<codeblock>jdbc:impala://<varname>Host</varname>:<varname>Port</varname>[/<varname>Schema</varname>];<varname>Property1</varname>=<varname>Value</varname>;<varname>Property2</varname>=<varname>Value</varname>;...</codeblock>
+
+        <p>
+          The <codeph>port</codeph> value is typically 21050 for Impala.
+        </p>
+
+        <p>
+          For full details about the classes and the connection string (especially the property values available
+          for the connection string), download the appropriate driver documentation for your platform from
+          <xref href="http://www.cloudera.com/content/cloudera/en/downloads/connectors/impala/jdbc/impala-jdbc-v2-5-5.html" scope="external" format="html">the Impala JDBC Connector download page</xref>.
+        </p>
+
+      </section>
+
+      <section id="class_hive_driver">
+      <title>Using the Hive JDBC Driver</title>
+
+      <p>
+        For example, with the Hive JDBC driver, the class name is <codeph>org.apache.hive.jdbc.HiveDriver</codeph>.
+        Once you have configured Impala to work with JDBC, you can establish connections between the two.
+        To do so for a cluster that does not use
+        Kerberos authentication, use a connection string of the form
+        <codeph>jdbc:hive2://<varname>host</varname>:<varname>port</varname>/;auth=noSasl</codeph>.
+<!--
+        Include the <codeph>auth=noSasl</codeph> argument
+        only when connecting to a non-Kerberos cluster; if Kerberos is enabled, omit the <codeph>auth</codeph> argument.
+-->
+        For example, you might use:
+      </p>
+
+<codeblock>jdbc:hive2://myhost.example.com:21050/;auth=noSasl</codeblock>
+
+      <p>
+        To connect to an instance of Impala that requires Kerberos authentication, use a connection string of the
+        form
+        <codeph>jdbc:hive2://<varname>host</varname>:<varname>port</varname>/;principal=<varname>principal_name</varname></codeph>.
+        The principal must be the same user principal you used when starting Impala. For example, you might use:
+      </p>
+
+<codeblock>jdbc:hive2://myhost.example.com:21050/;principal=impala/myhost.example.com@H2.EXAMPLE.COM</codeblock>
+
+      <p>
+        To connect to an instance of Impala that requires LDAP authentication, use a connection string of the form
+        <codeph>jdbc:hive2://<varname>host</varname>:<varname>port</varname>/<varname>db_name</varname>;user=<varname>ldap_userid</varname>;password=<varname>ldap_password</varname></codeph>.
+        For example, you might use:
+      </p>
+
+<codeblock>jdbc:hive2://myhost.example.com:21050/test_db;user=fred;password=xyz123</codeblock>
+
+      <note>
+          <p conref="../shared/impala_common.xml#common/hive_jdbc_ssl_kerberos_caveat"/>
+      </note>
+
+      </section>
+
+    </conbody>
+  </concept>
+
+  <concept rev="2.3.0" id="jdbc_odbc_notes">
+    <title>Notes about JDBC and ODBC Interaction with Impala SQL Features</title>
+    <conbody>
+      <p>
+        Most Impala SQL features work equivalently through the <cmdname>impala-shell</cmdname> interpreter
+        of the JDBC or ODBC APIs. The following are some exceptions to keep in mind when switching between
+        the interactive shell and applications using the APIs:
+      </p>
+      <ul>
+        <li>
+          <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+          <ul>
+          <li>
+          <p>
+            Queries involving the complex types (<codeph>ARRAY</codeph>, <codeph>STRUCT</codeph>, and <codeph>MAP</codeph>)
+            require notation that might not be available in all levels of JDBC and ODBC drivers.
+            If you have trouble querying such a table due to the driver level or
+            inability to edit the queries used by the application, you can create a view that exposes
+            a <q>flattened</q> version of the complex columns and point the application at the view.
+            See <xref href="impala_complex_types.xml#complex_types"/> for details.
+          </p>
+        </li>
+        <li>
+          <p>
+            The complex types available in <keyword keyref="impala23_full"/> and higher are supported by the
+            JDBC <codeph>getColumns()</codeph> API.
+            Both <codeph>MAP</codeph> and <codeph>ARRAY</codeph> are reported as the JDBC SQL Type <codeph>ARRAY</codeph>,
+            because this is the closest matching Java SQL type. This behavior is consistent with Hive.
+            <codeph>STRUCT</codeph> types are reported as the JDBC SQL Type <codeph>STRUCT</codeph>.
+          </p>
+          <p>
+            To be consistent with Hive's behavior, the TYPE_NAME field is populated
+            with the primitive type name for scalar types, and with the full <codeph>toSql()</codeph>
+            for complex types. The resulting type names are somewhat inconsistent,
+            because nested types are printed differently than top-level types. For example,
+            the following list shows how <codeph>toSQL()</codeph> for Impala types are
+            translated to <codeph>TYPE_NAME</codeph> values:
+<codeblock><![CDATA[DECIMAL(10,10)         becomes  DECIMAL
+CHAR(10)               becomes  CHAR
+VARCHAR(10)            becomes  VARCHAR
+ARRAY<DECIMAL(10,10)>  becomes  ARRAY<DECIMAL(10,10)>
+ARRAY<CHAR(10)>        becomes  ARRAY<CHAR(10)>
+ARRAY<VARCHAR(10)>     becomes  ARRAY<VARCHAR(10)>
+]]>
+</codeblock>
+          </p>
+          </li>
+        </ul>
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+</concept>


[05/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_string.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_string.xml b/docs/topics/impala_string.xml
new file mode 100644
index 0000000..1be9d98
--- /dev/null
+++ b/docs/topics/impala_string.xml
@@ -0,0 +1,180 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="string">
+
+  <title>STRING Data Type</title>
+  <titlealts audience="PDF"><navtitle>STRING</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      A data type used in <codeph>CREATE TABLE</codeph> and <codeph>ALTER TABLE</codeph> statements.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p>
+      In the column definition of a <codeph>CREATE TABLE</codeph> statement:
+    </p>
+
+<codeblock><varname>column_name</varname> STRING</codeblock>
+
+    <p>
+      <b>Length:</b> Maximum of 32,767 bytes. Do not use any length constraint when declaring
+      <codeph>STRING</codeph> columns, as you might be familiar with from <codeph>VARCHAR</codeph>,
+      <codeph>CHAR</codeph>, or similar column types from relational database systems. <ph rev="2.0.0">If you do
+      need to manipulate string values with precise or maximum lengths, in Impala 2.0 and higher you can declare
+      columns as <codeph>VARCHAR(<varname>max_length</varname>)</codeph> or
+      <codeph>CHAR(<varname>length</varname>)</codeph>, but for best performance use <codeph>STRING</codeph>
+      where practical.</ph>
+    </p>
+
+    <p>
+      <b>Character sets:</b> For full support in all Impala subsystems, restrict string values to the ASCII
+      character set. Although some UTF-8 character data can be stored in Impala and retrieved through queries, UTF-8 strings
+      containing non-ASCII characters are not guaranteed to work properly in combination with many SQL aspects,
+      including but not limited to:
+    </p>
+    <ul>
+      <li>
+        String manipulation functions.
+      </li>
+      <li>
+        Comparison operators.
+      </li>
+      <li>
+        The <codeph>ORDER BY</codeph> clause.
+      </li>
+      <li>
+        Values in partition key columns.
+      </li>
+    </ul>
+
+    <p>
+      For any national language aspects such as
+      collation order or interpreting extended ASCII variants such as ISO-8859-1 or ISO-8859-2 encodings, Impala
+      does not include such metadata with the table definition. If you need to sort, manipulate, or display data
+      depending on those national language characteristics of string data, use logic on the application side.
+    </p>
+
+    <p>
+      <b>Conversions:</b>
+    </p>
+
+    <ul>
+      <li>
+        <p>
+          Impala does not automatically convert <codeph>STRING</codeph> to any numeric type. Impala does
+          automatically convert <codeph>STRING</codeph> to <codeph>TIMESTAMP</codeph> if the value matches one of
+          the accepted <codeph>TIMESTAMP</codeph> formats; see <xref href="impala_timestamp.xml#timestamp"/> for
+          details.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          You can use <codeph>CAST()</codeph> to convert <codeph>STRING</codeph> values to
+          <codeph>TINYINT</codeph>, <codeph>SMALLINT</codeph>, <codeph>INT</codeph>, <codeph>BIGINT</codeph>,
+          <codeph>FLOAT</codeph>, <codeph>DOUBLE</codeph>, or <codeph>TIMESTAMP</codeph>.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          You cannot directly cast a <codeph>STRING</codeph> value to <codeph>BOOLEAN</codeph>. You can use a
+          <codeph>CASE</codeph> expression to evaluate string values such as <codeph>'T'</codeph>,
+          <codeph>'true'</codeph>, and so on and return Boolean <codeph>true</codeph> and <codeph>false</codeph>
+          values as appropriate.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          You can cast a <codeph>BOOLEAN</codeph> value to <codeph>STRING</codeph>, returning <codeph>'1'</codeph>
+          for <codeph>true</codeph> values and <codeph>'0'</codeph> for <codeph>false</codeph> values.
+        </p>
+      </li>
+    </ul>
+
+    <p conref="../shared/impala_common.xml#common/partitioning_blurb"/>
+
+    <p>
+      Although it might be convenient to use <codeph>STRING</codeph> columns for partition keys, even when those
+      columns contain numbers, for performance and scalability it is much better to use numeric columns as
+      partition keys whenever practical. Although the underlying HDFS directory name might be the same in either
+      case, the in-memory storage for the partition key columns is more compact, and computations are faster, if
+      partition key columns such as <codeph>YEAR</codeph>, <codeph>MONTH</codeph>, <codeph>DAY</codeph> and so on
+      are declared as <codeph>INT</codeph>, <codeph>SMALLINT</codeph>, and so on.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/zero_length_strings"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/hbase_blurb"/> -->
+
+<!-- <p conref="../shared/impala_common.xml#common/parquet_blurb"/> -->
+
+    <p conref="../shared/impala_common.xml#common/text_bulky"/>
+
+    <p><b>Avro considerations:</b></p>
+    <p conref="../shared/impala_common.xml#common/avro_2gb_strings"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/compatibility_blurb"/> -->
+
+<!-- <p conref="../shared/impala_common.xml#common/internals_blurb"/> -->
+
+<!-- <p conref="../shared/impala_common.xml#common/added_in_20"/> -->
+
+    <p conref="../shared/impala_common.xml#common/column_stats_variable"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      The following examples demonstrate double-quoted and single-quoted string literals, and required escaping for
+      quotation marks within string literals:
+    </p>
+
+<codeblock>SELECT 'I am a single-quoted string';
+SELECT "I am a double-quoted string";
+SELECT 'I\'m a single-quoted string with an apostrophe';
+SELECT "I\'m a double-quoted string with an apostrophe";
+SELECT 'I am a "short" single-quoted string containing quotes';
+SELECT "I am a \"short\" double-quoted string containing quotes";
+</codeblock>
+
+    <p>
+      The following examples demonstrate calls to string manipulation functions to concatenate strings, convert
+      numbers to strings, or pull out substrings:
+    </p>
+
+<codeblock>SELECT CONCAT("Once upon a time, there were ", CAST(3 AS STRING), ' little pigs.');
+SELECT SUBSTR("hello world",7,5);
+</codeblock>
+
+    <p>
+      The following examples show how to perform operations on <codeph>STRING</codeph> columns within a table:
+    </p>
+
+<codeblock>CREATE TABLE t1 (s1 STRING, s2 STRING);
+INSERT INTO t1 VALUES ("hello", 'world'), (CAST(7 AS STRING), "wonders");
+SELECT s1, s2, length(s1) FROM t1 WHERE s2 LIKE 'w%';
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_literals.xml#string_literals"/>, <xref href="impala_char.xml#char"/>,
+      <xref href="impala_varchar.xml#varchar"/>, <xref href="impala_string_functions.xml#string_functions"/>,
+      <xref href="impala_datetime_functions.xml#datetime_functions"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_string_functions.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_string_functions.xml b/docs/topics/impala_string_functions.xml
new file mode 100644
index 0000000..8f8636f
--- /dev/null
+++ b/docs/topics/impala_string_functions.xml
@@ -0,0 +1,926 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="string_functions">
+
+  <title>Impala String Functions</title>
+  <titlealts audience="PDF"><navtitle>String Functions</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Querying"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.0.0">
+      String functions are classified as those primarily accepting or returning <codeph>STRING</codeph>,
+      <codeph>VARCHAR</codeph>, or <codeph>CHAR</codeph> data types, for example to measure the length of a string
+      or concatenate two strings together.
+      <ul>
+        <li>
+          All the functions that accept <codeph>STRING</codeph> arguments also accept the <codeph>VARCHAR</codeph>
+          and <codeph>CHAR</codeph> types introduced in Impala 2.0.
+        </li>
+
+        <li>
+          Whenever <codeph>VARCHAR</codeph> or <codeph>CHAR</codeph> values are passed to a function that returns a
+          string value, the return type is normalized to <codeph>STRING</codeph>. For example, a call to
+          <codeph>concat()</codeph> with a mix of <codeph>STRING</codeph>, <codeph>VARCHAR</codeph>, and
+          <codeph>CHAR</codeph> arguments produces a <codeph>STRING</codeph> result.
+        </li>
+      </ul>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      The string functions operate mainly on these data types: <xref href="impala_string.xml#string"/>,
+      <xref href="impala_varchar.xml#varchar"/>, and <xref href="impala_char.xml#char"/>.
+    </p>
+
+    <p>
+      <b>Function reference:</b>
+    </p>
+
+    <p>
+      Impala supports the following string functions:
+    </p>
+
+    <dl>
+      <dlentry id="ascii">
+
+        <dt>
+          <codeph>ascii(string str)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">ascii() function</indexterm>
+          <b>Purpose:</b> Returns the numeric ASCII code of the first character of the argument.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="2.3.0" id="btrim">
+
+        <dt>
+          <codeph>btrim(string a)</codeph>,
+          <codeph>btrim(string a, string chars_to_trim)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">btrim() function</indexterm>
+          <b>Purpose:</b> Removes all instances of one or more characters
+          from the start and end of a <codeph>STRING</codeph> value.
+          By default, removes only spaces.
+          If a non-<codeph>NULL</codeph> optional second argument is specified, the function removes all
+          occurrences of characters in that second argument from the beginning and
+          end of the string.
+          <p><b>Return type:</b> <codeph>string</codeph></p>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples show the default <codeph>btrim()</codeph> behavior,
+            and what changes when you specify the optional second argument.
+            All the examples bracket the output value with <codeph>[ ]</codeph>
+            so that you can see any leading or trailing spaces in the <codeph>btrim()</codeph> result.
+            By default, the function removes and number of both leading and trailing spaces.
+            When the second argument is specified, any number of occurrences of any
+            character in the second argument are removed from the start and end of the
+            input string; in this case, spaces are not removed (unless they are part of the second
+            argument) and any instances of the characters are not removed if they do not come
+            right at the beginning or end of the string.
+          </p>
+<codeblock>-- Remove multiple spaces before and one space after.
+select concat('[',btrim('    hello '),']');
++---------------------------------------+
+| concat('[', btrim('    hello '), ']') |
++---------------------------------------+
+| [hello]                               |
++---------------------------------------+
+
+-- Remove any instances of x or y or z at beginning or end. Leave spaces alone.
+select concat('[',btrim('xy    hello zyzzxx','xyz'),']');
++------------------------------------------------------+
+| concat('[', btrim('xy    hello zyzzxx', 'xyz'), ']') |
++------------------------------------------------------+
+| [    hello ]                                         |
++------------------------------------------------------+
+
+-- Remove any instances of x or y or z at beginning or end.
+-- Leave x, y, z alone in the middle of the string.
+select concat('[',btrim('xyhelxyzlozyzzxx','xyz'),']');
++----------------------------------------------------+
+| concat('[', btrim('xyhelxyzlozyzzxx', 'xyz'), ']') |
++----------------------------------------------------+
+| [helxyzlo]                                         |
++----------------------------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="char_length">
+
+        <dt>
+          <codeph>char_length(string a), <ph rev="1.3.0" id="character_length">character_length(string a)</ph></codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">char_length() function</indexterm>
+          <indexterm audience="Cloudera">character_length() function</indexterm>
+          <b>Purpose:</b> Returns the length in characters of the argument string. Aliases for the
+          <codeph>length()</codeph> function.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="2.3.0" id="chr">
+
+        <dt>
+          <codeph>chr(int character_code)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">chr() function</indexterm>
+          <b>Purpose:</b> Returns a character specified by a decimal code point value.
+          The interpretation and display of the resulting character depends on your system locale.
+          Because consistent processing of Impala string values is only guaranteed
+          for values within the ASCII range, only use this function for values
+          corresponding to ASCII characters.
+          In particular, parameter values greater than 255 return an empty string.
+          <p><b>Return type:</b> <codeph>string</codeph></p>
+          <p>
+            <b>Usage notes:</b> Can be used as the inverse of the <codeph>ascii()</codeph> function, which
+            converts a character to its numeric ASCII code.
+          </p>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>SELECT chr(65);
++---------+
+| chr(65) |
++---------+
+| A       |
++---------+
+
+SELECT chr(97);
++---------+
+| chr(97) |
++---------+
+| a       |
++---------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="concat">
+
+        <dt>
+          <codeph>concat(string a, string b...)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">concat() function</indexterm>
+          <b>Purpose:</b> Returns a single string representing all the argument values joined together.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/concat_blurb"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="concat_ws">
+
+        <dt>
+          <codeph>concat_ws(string sep, string a, string b...)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">concat_ws() function</indexterm>
+          <b>Purpose:</b> Returns a single string representing the second and following argument values joined
+          together, delimited by a specified separator.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/concat_blurb"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="find_in_set">
+
+        <dt>
+          <codeph>find_in_set(string str, string strList)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">find_in_set() function</indexterm>
+          <b>Purpose:</b> Returns the position (starting from 1) of the first occurrence of a specified string
+          within a comma-separated string. Returns <codeph>NULL</codeph> if either argument is
+          <codeph>NULL</codeph>, 0 if the search string is not found, or 0 if the search string contains a comma.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.2" id="group_concat">
+
+        <dt>
+          <codeph>group_concat(string s [, string sep])</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">group_concat() function</indexterm>
+          <b>Purpose:</b> Returns a single string representing the argument value concatenated together for each
+          row of the result set. If the optional separator string is specified, the separator is added between each
+          pair of concatenated values.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/concat_blurb"/>
+          <p>
+            By default, returns a single string covering the whole result set. To include other columns or values
+            in the result set, or to produce multiple concatenated strings for subsets of rows, include a
+            <codeph>GROUP BY</codeph> clause in the query.
+          </p>
+          <p>
+            Strictly speaking, <codeph>group_concat()</codeph> is an aggregate function, not a scalar
+            function like the others in this list.
+            For additional details and examples, see <xref href="impala_group_concat.xml#group_concat"/>.
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.2" id="initcap">
+
+        <dt>
+          <codeph>initcap(string str)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">initcap() function</indexterm>
+          <b>Purpose:</b> Returns the input string with the first letter capitalized.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="instr">
+
+        <dt>
+          <codeph>instr(string str, string substr)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">instr() function</indexterm>
+          <b>Purpose:</b> Returns the position (starting from 1) of the first occurrence of a substring within a
+          longer string.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="length">
+
+        <dt>
+          <codeph>length(string a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">length() function</indexterm>
+          <b>Purpose:</b> Returns the length in characters of the argument string.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="locate">
+
+        <dt>
+          <codeph>locate(string substr, string str[, int pos])</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">locate() function</indexterm>
+          <b>Purpose:</b> Returns the position (starting from 1) of the first occurrence of a substring within a
+          longer string, optionally after a particular position.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="lower">
+
+        <dt>
+          <codeph>lower(string a), <ph id="lcase">lcase(string a)</ph> </codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">lower() function</indexterm>
+          <b>Purpose:</b> Returns the argument string converted to all-lowercase.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p conref="../shared/impala_common.xml#common/case_insensitive_comparisons_tip"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="lpad">
+
+        <dt>
+          <codeph>lpad(string str, int len, string pad)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">lpad() function</indexterm>
+          <b>Purpose:</b> Returns a string of a specified length, based on the first argument string. If the
+          specified string is too short, it is padded on the left with a repeating sequence of the characters from
+          the pad string. If the specified string is too long, it is truncated on the right.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="ltrim">
+
+        <dt>
+          <codeph>ltrim(string a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">ltrim() function</indexterm>
+          <b>Purpose:</b> Returns the argument string with any leading spaces removed from the left side.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="parse_url">
+
+        <dt>
+          <codeph>parse_url(string urlString, string partToExtract [, string keyToExtract])</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">parse_url() function</indexterm>
+          <b>Purpose:</b> Returns the portion of a URL corresponding to a specified part. The part argument can be
+          <codeph>'PROTOCOL'</codeph>, <codeph>'HOST'</codeph>, <codeph>'PATH'</codeph>, <codeph>'REF'</codeph>,
+          <codeph>'AUTHORITY'</codeph>, <codeph>'FILE'</codeph>, <codeph>'USERINFO'</codeph>, or
+          <codeph>'QUERY'</codeph>. Uppercase is required for these literal values. When requesting the
+          <codeph>QUERY</codeph> portion of the URL, you can optionally specify a key to retrieve just the
+          associated value from the key-value pairs in the query string.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+          <p>
+            <b>Usage notes:</b> This function is important for the traditional Hadoop use case of interpreting web
+            logs. For example, if the web traffic data features raw URLs not divided into separate table columns,
+            you can count visitors to a particular page by extracting the <codeph>'PATH'</codeph> or
+            <codeph>'FILE'</codeph> field, or analyze search terms by extracting the corresponding key from the
+            <codeph>'QUERY'</codeph> field.
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="regexp_extract">
+
+        <dt>
+          <codeph>regexp_extract(string subject, string pattern, int index)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">regexp_extract() function</indexterm>
+          <b>Purpose:</b> Returns the specified () group from a string based on a regular expression pattern. Group
+          0 refers to the entire extracted string, while group 1, 2, and so on refers to the first, second, and so
+          on <codeph>(...)</codeph> portion.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/regexp_re2"/>
+          <p conref="../shared/impala_common.xml#common/regexp_re2_warning"/>
+          <p conref="../shared/impala_common.xml#common/regexp_escapes"/>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            This example shows how group 0 matches the full pattern string, including the portion outside any
+            <codeph>()</codeph> group:
+          </p>
+<codeblock>[localhost:21000] &gt; select regexp_extract('abcdef123ghi456jkl','.*?(\\d+)',0);
++------------------------------------------------------+
+| regexp_extract('abcdef123ghi456jkl', '.*?(\\d+)', 0) |
++------------------------------------------------------+
+| abcdef123ghi456                                      |
++------------------------------------------------------+
+Returned 1 row(s) in 0.11s</codeblock>
+          <p>
+            This example shows how group 1 matches just the contents inside the first <codeph>()</codeph> group in
+            the pattern string:
+          </p>
+<codeblock>[localhost:21000] &gt; select regexp_extract('abcdef123ghi456jkl','.*?(\\d+)',1);
++------------------------------------------------------+
+| regexp_extract('abcdef123ghi456jkl', '.*?(\\d+)', 1) |
++------------------------------------------------------+
+| 456                                                  |
++------------------------------------------------------+
+Returned 1 row(s) in 0.11s</codeblock>
+          <p rev="2.0.0">
+            Unlike in earlier Impala releases, the regular expression library used in Impala 2.0 and later supports
+            the <codeph>.*?</codeph> idiom for non-greedy matches. This example shows how a pattern string starting
+            with <codeph>.*?</codeph> matches the shortest possible portion of the source string, returning the
+            rightmost set of lowercase letters. A pattern string both starting and ending with <codeph>.*?</codeph>
+            finds two potential matches of equal length, and returns the first one found (the leftmost set of
+            lowercase letters).
+          </p>
+<codeblock>[localhost:21000] &gt; select regexp_extract('AbcdBCdefGHI','.*?([[:lower:]]+)',1);
++--------------------------------------------------------+
+| regexp_extract('abcdbcdefghi', '.*?([[:lower:]]+)', 1) |
++--------------------------------------------------------+
+| def                                                    |
++--------------------------------------------------------+
+[localhost:21000] &gt; select regexp_extract('AbcdBCdefGHI','.*?([[:lower:]]+).*?',1);
++-----------------------------------------------------------+
+| regexp_extract('abcdbcdefghi', '.*?([[:lower:]]+).*?', 1) |
++-----------------------------------------------------------+
+| bcd                                                       |
++-----------------------------------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="2.3.0" id="regexp_like">
+
+        <dt>
+          <codeph>regexp_like(string source, string pattern[, string options])</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">regexp_like() function</indexterm>
+          <b>Purpose:</b> Returns <codeph>true</codeph> or <codeph>false</codeph> to indicate
+          whether the source string contains anywhere inside it the regular expression given by the pattern.
+          The optional third argument consists of letter flags that change how the match is performed,
+          such as <codeph>i</codeph> for case-insensitive matching.
+          <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+          <p>
+            The flags that you can include in the optional third argument are:
+          </p>
+          <ul>
+          <li>
+          <codeph>c</codeph>: Case-sensitive matching (the default).
+          </li>
+          <li>
+          <codeph>i</codeph>: Case-insensitive matching. If multiple instances of <codeph>c</codeph> and <codeph>i</codeph>
+          are included in the third argument, the last such option takes precedence.
+          </li>
+          <li>
+          <codeph>m</codeph>: Multi-line matching. The <codeph>^</codeph> and <codeph>$</codeph>
+          operators match the start or end of any line within the source string, not the
+          start and end of the entire string.
+          </li>
+          <li>
+          <codeph>n</codeph>: Newline matching. The <codeph>.</codeph> operator can match the
+          newline character. A repetition operator such as <codeph>.*</codeph> can
+          match a portion of the source string that spans multiple lines.
+          </li>
+          </ul>
+          <p>
+            <b>Return type:</b> <codeph>boolean</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/regexp_re2"/>
+          <p conref="../shared/impala_common.xml#common/regexp_re2_warning"/>
+          <p conref="../shared/impala_common.xml#common/regexp_escapes"/>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            This example shows how <codeph>regexp_like()</codeph> can test for the existence
+            of various kinds of regular expression patterns within a source string:
+          </p>
+<codeblock><![CDATA[
+-- Matches because the 'f' appears somewhere in 'foo'.
+select regexp_like('foo','f');
++-------------------------+
+| regexp_like('foo', 'f') |
++-------------------------+
+| true                    |
++-------------------------+
+
+-- Does not match because the comparison is case-sensitive by default.
+select regexp_like('foo','F');
++-------------------------+
+| regexp_like('foo', 'f') |
++-------------------------+
+| false                   |
++-------------------------+
+
+-- The 3rd argument can change the matching logic, such as 'i' meaning case-insensitive.
+select regexp_like('foo','F','i');
++------------------------------+
+| regexp_like('foo', 'f', 'i') |
++------------------------------+
+| true                         |
++------------------------------+
+
+-- The familiar regular expression notations work, such as ^ and $ anchors...
+select regexp_like('foo','f$');
++--------------------------+
+| regexp_like('foo', 'f$') |
++--------------------------+
+| false                    |
++--------------------------+
+
+select regexp_like('foo','o$');
++--------------------------+
+| regexp_like('foo', 'o$') |
++--------------------------+
+| true                     |
++--------------------------+
+
+-- ...and repetition operators such as * and +
+select regexp_like('foooooobar','fo+b');
++-----------------------------------+
+| regexp_like('foooooobar', 'fo+b') |
++-----------------------------------+
+| true                              |
++-----------------------------------+
+
+select regexp_like('foooooobar','fx*y*o*b');
++---------------------------------------+
+| regexp_like('foooooobar', 'fx*y*o*b') |
++---------------------------------------+
+| true                                  |
++---------------------------------------+
+]]>
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="regexp_replace">
+
+        <dt>
+          <codeph>regexp_replace(string initial, string pattern, string replacement)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">regexp_replace() function</indexterm>
+          <b>Purpose:</b> Returns the initial argument with the regular expression pattern replaced by the final
+          argument string.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/regexp_re2"/>
+          <p conref="../shared/impala_common.xml#common/regexp_re2_warning"/>
+          <p conref="../shared/impala_common.xml#common/regexp_escapes"/>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            These examples show how you can replace parts of a string matching a pattern with replacement text,
+            which can include backreferences to any <codeph>()</codeph> groups in the pattern string. The
+            backreference numbers start at 1, and any <codeph>\</codeph> characters must be escaped as
+            <codeph>\\</codeph>.
+          </p>
+          <p>
+            Replace a character pattern with new text:
+          </p>
+<codeblock>[localhost:21000] &gt; select regexp_replace('aaabbbaaa','b+','xyz');
++------------------------------------------+
+| regexp_replace('aaabbbaaa', 'b+', 'xyz') |
++------------------------------------------+
+| aaaxyzaaa                                |
++------------------------------------------+
+Returned 1 row(s) in 0.11s</codeblock>
+          <p>
+            Replace a character pattern with substitution text that includes the original matching text:
+          </p>
+<codeblock>[localhost:21000] &gt; select regexp_replace('aaabbbaaa','(b+)','&lt;\\1&gt;');
++----------------------------------------------+
+| regexp_replace('aaabbbaaa', '(b+)', '&lt;\\1&gt;') |
++----------------------------------------------+
+| aaa&lt;bbb&gt;aaa                                  |
++----------------------------------------------+
+Returned 1 row(s) in 0.11s</codeblock>
+          <p>
+            Remove all characters that are not digits:
+          </p>
+<codeblock>[localhost:21000] &gt; select regexp_replace('123-456-789','[^[:digit:]]','');
++---------------------------------------------------+
+| regexp_replace('123-456-789', '[^[:digit:]]', '') |
++---------------------------------------------------+
+| 123456789                                         |
++---------------------------------------------------+
+Returned 1 row(s) in 0.12s</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="repeat">
+
+        <dt>
+          <codeph>repeat(string str, int n)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">repeat() function</indexterm>
+          <b>Purpose:</b> Returns the argument string repeated a specified number of times.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="reverse">
+
+        <dt>
+          <codeph>reverse(string a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">reverse() function</indexterm>
+          <b>Purpose:</b> Returns the argument string with characters in reversed order.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="rpad">
+
+        <dt>
+          <codeph>rpad(string str, int len, string pad)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">rpad() function</indexterm>
+          <b>Purpose:</b> Returns a string of a specified length, based on the first argument string. If the
+          specified string is too short, it is padded on the right with a repeating sequence of the characters from
+          the pad string. If the specified string is too long, it is truncated on the right.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="rtrim">
+
+        <dt>
+          <codeph>rtrim(string a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">rtrim() function</indexterm>
+          <b>Purpose:</b> Returns the argument string with any trailing spaces removed from the right side.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="space">
+
+        <dt>
+          <codeph>space(int n)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">space() function</indexterm>
+          <b>Purpose:</b> Returns a concatenated string of the specified number of spaces. Shorthand for
+          <codeph>repeat(' ',<varname>n</varname>)</codeph>.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="2.3.0 CDH-35915 IMPALA-2084" id="split_part">
+
+        <dt>
+          <codeph>split_part(string source, string delimiter, bigint n)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">split_part() function</indexterm>
+          <b>Purpose:</b> Returns the nth field within a delimited string.
+          The fields are numbered starting from 1.
+          The delimiter can consist of multiple characters, not just a
+          single character. All matching of the delimiter is done exactly, not using any
+          regular expression patterns.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/regexp_re2"/>
+          <p conref="../shared/impala_common.xml#common/regexp_re2_warning"/>
+          <p conref="../shared/impala_common.xml#common/regexp_escapes"/>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            These examples show how to retrieve the nth field from a delimited string:
+          </p>
+<codeblock><![CDATA[
+select split_part('x,y,z',',',1);
++-----------------------------+
+| split_part('x,y,z', ',', 1) |
++-----------------------------+
+| x                           |
++-----------------------------+
+
+select split_part('x,y,z',',',2);
++-----------------------------+
+| split_part('x,y,z', ',', 2) |
++-----------------------------+
+| y                           |
++-----------------------------+
+
+select split_part('x,y,z',',',3);
++-----------------------------+
+| split_part('x,y,z', ',', 3) |
++-----------------------------+
+| z                           |
++-----------------------------+
+</codeblock>
+
+          <p>
+            These examples show what happens for out-of-range field positions.
+            Specifying a value less than 1 produces an error. Specifying a value
+            greater than the number of fields returns a zero-length string
+            (which is not the same as <codeph>NULL</codeph>).
+          </p>
+<codeblock><![CDATA[
+select split_part('x,y,z',',',0);
+ERROR: Invalid field position: 0
+
+with t1 as (select split_part('x,y,z',',',4) nonexistent_field)
+  select
+      nonexistent_field
+    , concat('[',nonexistent_field,']')
+    , length(nonexistent_field);
+from t1
++-------------------+-------------------------------------+---------------------------+
+| nonexistent_field | concat('[', nonexistent_field, ']') | length(nonexistent_field) |
++-------------------+-------------------------------------+---------------------------+
+|                   | []                                  | 0                         |
++-------------------+-------------------------------------+---------------------------+
+</codeblock>
+
+          <p>
+            These examples show how the delimiter can be a multi-character value:
+          </p>
+<codeblock><![CDATA[
+select split_part('one***two***three','***',2);
++-------------------------------------------+
+| split_part('one***two***three', '***', 2) |
++-------------------------------------------+
+| two                                       |
++-------------------------------------------+
+
+select split_part('one\|/two\|/three','\|/',3);
++-------------------------------------------+
+| split_part('one\|/two\|/three', '\|/', 3) |
++-------------------------------------------+
+| three                                     |
++-------------------------------------------+
+]]>
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="strleft">
+
+        <dt>
+          <codeph>strleft(string a, int num_chars)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">strleft() function</indexterm>
+          <b>Purpose:</b> Returns the leftmost characters of the string. Shorthand for a call to
+          <codeph>substr()</codeph> with 2 arguments.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="strright">
+
+        <dt>
+          <codeph>strright(string a, int num_chars)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">strright() function</indexterm>
+          <b>Purpose:</b> Returns the rightmost characters of the string. Shorthand for a call to
+          <codeph>substr()</codeph> with 2 arguments.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="substr">
+
+        <dt>
+          <codeph>substr(string a, int start [, int len]), <ph id="substring">substring(string a, int start [, int
+          len])</ph></codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">substr() function</indexterm>
+          <b>Purpose:</b> Returns the portion of the string starting at a specified point, optionally with a
+          specified maximum length. The characters in the string are indexed starting at 1.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="translate">
+
+        <dt>
+          <codeph>translate(string input, string from, string to)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">translate() function</indexterm>
+          <b>Purpose:</b> Returns the input string with a set of characters replaced by another set of characters.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="trim">
+
+        <dt>
+          <codeph>trim(string a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">trim() function</indexterm>
+          <b>Purpose:</b> Returns the input string with both leading and trailing spaces removed. The same as
+          passing the string through both <codeph>ltrim()</codeph> and <codeph>rtrim()</codeph>.
+          <p>
+            <b>Usage notes:</b> Often used during data cleansing operations during the ETL cycle, if input values might still have surrounding spaces.
+            For a more general-purpose function that can remove other leading and trailing characters besides spaces, see <codeph>btrim()</codeph>.
+          </p>
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="upper">
+
+        <dt>
+          <codeph>upper(string a), <ph id="ucase">ucase(string a)</ph></codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">upper() function</indexterm>
+          <indexterm audience="Cloudera">ucase() function</indexterm>
+          <b>Purpose:</b> Returns the argument string converted to all-uppercase.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p conref="../shared/impala_common.xml#common/case_insensitive_comparisons_tip"/>
+        </dd>
+
+      </dlentry>
+    </dl>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_struct.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_struct.xml b/docs/topics/impala_struct.xml
new file mode 100644
index 0000000..bddb1bb
--- /dev/null
+++ b/docs/topics/impala_struct.xml
@@ -0,0 +1,433 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="struct">
+
+  <title>STRUCT Complex Type (<keyword keyref="impala23"/> or higher only)</title>
+
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      A complex data type, representing multiple fields of a single item. Frequently used as the element type of an <codeph>ARRAY</codeph>
+      or the <codeph>VALUE</codeph> part of a <codeph>MAP</codeph>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock><varname>column_name</varname> STRUCT &lt; <varname>name</varname> : <varname>type</varname> [COMMENT '<varname>comment_string</varname>'], ... &gt;
+
+type ::= <varname>primitive_type</varname> | <varname>complex_type</varname>
+</codeblock>
+
+    <p>
+      The names and number of fields within the <codeph>STRUCT</codeph> are fixed. Each field can be a different type. A field within a
+      <codeph>STRUCT</codeph> can also be another <codeph>STRUCT</codeph>, or an <codeph>ARRAY</codeph> or a <codeph>MAP</codeph>, allowing
+      you to create nested data structures with a maximum nesting depth of 100.
+    </p>
+
+    <p>
+      A <codeph>STRUCT</codeph> can be the top-level type for a column, or can itself be an item within an <codeph>ARRAY</codeph> or the
+      value part of the key-value pair in a <codeph>MAP</codeph>.
+    </p>
+
+    <p>
+      When a <codeph>STRUCT</codeph> is used as an <codeph>ARRAY</codeph> element or a <codeph>MAP</codeph> value, you use a join clause to
+      bring the <codeph>ARRAY</codeph> or <codeph>MAP</codeph> elements into the result set, and then refer to
+      <codeph><varname>array_name</varname>.ITEM.<varname>field</varname></codeph> or
+      <codeph><varname>map_name</varname>.VALUE.<varname>field</varname></codeph>. In the case of a <codeph>STRUCT</codeph> directly inside
+      an <codeph>ARRAY</codeph> or <codeph>MAP</codeph>, you can omit the <codeph>.ITEM</codeph> and <codeph>.VALUE</codeph> pseudocolumns
+      and refer directly to <codeph><varname>array_name</varname>.<varname>field</varname></codeph> or
+      <codeph><varname>map_name</varname>.<varname>field</varname></codeph>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_combo"/>
+
+    <p>
+      A <codeph>STRUCT</codeph> is similar conceptually to a table row: it contains a fixed number of named fields, each with a predefined
+      type. To combine two related tables, while using complex types to minimize repetition, the typical way to represent that data is as an
+      <codeph>ARRAY</codeph> of <codeph>STRUCT</codeph> elements.
+    </p>
+
+    <p>
+      Because a <codeph>STRUCT</codeph> has a fixed number of named fields, it typically does not make sense to have a
+      <codeph>STRUCT</codeph> as the type of a table column. In such a case, you could just make each field of the <codeph>STRUCT</codeph>
+      into a separate column of the table. The <codeph>STRUCT</codeph> type is most useful as an item of an <codeph>ARRAY</codeph> or the
+      value part of the key-value pair in a <codeph>MAP</codeph>. A nested type column with a <codeph>STRUCT</codeph> at the lowest level
+      lets you associate a variable number of row-like objects with each row of the table.
+    </p>
+
+    <p>
+      The <codeph>STRUCT</codeph> type is straightforward to reference within a query. You do not need to include the
+      <codeph>STRUCT</codeph> column in a join clause or give it a table alias, as is required for the <codeph>ARRAY</codeph> and
+      <codeph>MAP</codeph> types. You refer to the individual fields using dot notation, such as
+      <codeph><varname>struct_column_name</varname>.<varname>field_name</varname></codeph>, without any pseudocolumn such as
+      <codeph>ITEM</codeph> or <codeph>VALUE</codeph>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_describe"/>
+
+    <p conref="../shared/impala_common.xml#common/internals_blurb"/>
+
+    <p>
+      Within the Parquet data file, the values for each <codeph>STRUCT</codeph> field are stored adjacent to each other, so that they can be
+      encoded and compressed using all the Parquet techniques for storing sets of similar or repeated values. The adjacency applies even
+      when the <codeph>STRUCT</codeph> values are part of an <codeph>ARRAY</codeph> or <codeph>MAP</codeph>. During a query, Impala avoids
+      unnecessary I/O by reading only the portions of the Parquet data file containing the requested <codeph>STRUCT</codeph> fields.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/added_in_230"/>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+    <ul conref="../shared/impala_common.xml#common/complex_types_restrictions">
+      <li/>
+    </ul>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <note conref="../shared/impala_common.xml#common/complex_type_schema_pointer"/>
+
+    <p>
+      The following example shows a table with various kinds of <codeph>STRUCT</codeph> columns, both at the top level and nested within
+      other complex types. Practice the <codeph>CREATE TABLE</codeph> and query notation for complex type columns using empty tables, until
+      you can visualize a complex data structure and construct corresponding SQL statements reliably.
+    </p>
+
+<codeblock><![CDATA[CREATE TABLE struct_demo
+(
+  id BIGINT,
+  name STRING,
+
+-- A STRUCT as a top-level column. Demonstrates how the table ID column
+-- and the ID field within the STRUCT can coexist without a name conflict.
+  employee_info STRUCT < employer: STRING, id: BIGINT, address: STRING >,
+
+-- A STRUCT as the element type of an ARRAY.
+  places_lived ARRAY < STRUCT <street: STRING, city: STRING, country: STRING >>,
+
+-- A STRUCT as the value portion of the key-value pairs in a MAP.
+  memorable_moments MAP < STRING, STRUCT < year: INT, place: STRING, details: STRING >>,
+
+-- A STRUCT where one of the fields is another STRUCT.
+  current_address STRUCT < street_address: STRUCT <street_number: INT, street_name: STRING, street_type: STRING>, country: STRING, postal_code: STRING >
+)
+STORED AS PARQUET;
+]]>
+</codeblock>
+
+    <p>
+      The following example shows how to examine the structure of a table containing one or more <codeph>STRUCT</codeph> columns by using
+      the <codeph>DESCRIBE</codeph> statement. You can visualize each <codeph>STRUCT</codeph> as its own table, with columns named the same
+      as each field of the <codeph>STRUCT</codeph>. If the <codeph>STRUCT</codeph> is nested inside another complex type, such as
+      <codeph>ARRAY</codeph>, you can extend the qualified name passed to <codeph>DESCRIBE</codeph> until the output shows just the
+      <codeph>STRUCT</codeph> fields.
+    </p>
+
+<codeblock><![CDATA[DESCRIBE struct_demo;
++-------------------+--------------------------+
+| name              | type                     |
++-------------------+--------------------------+
+| id                | bigint                   |
+| name              | string                   |
+| employee_info     | struct<                  |
+|                   |   employer:string,       |
+|                   |   id:bigint,             |
+|                   |   address:string         |
+|                   | >                        |
+| places_lived      | array<struct<            |
+|                   |   street:string,         |
+|                   |   city:string,           |
+|                   |   country:string         |
+|                   | >>                       |
+| memorable_moments | map<string,struct<       |
+|                   |   year:int,              |
+|                   |   place:string,          |
+|                   |   details:string         |
+|                   | >>                       |
+| current_address   | struct<                  |
+|                   |   street_address:struct< |
+|                   |     street_number:int,   |
+|                   |     street_name:string,  |
+|                   |     street_type:string   |
+|                   |   >,                     |
+|                   |   country:string,        |
+|                   |   postal_code:string     |
+|                   | >                        |
++-------------------+--------------------------+
+]]>
+</codeblock>
+
+    <p>
+      The top-level column <codeph>EMPLOYEE_INFO</codeph> is a <codeph>STRUCT</codeph>. Describing
+      <codeph><varname>table_name</varname>.<varname>struct_name</varname></codeph> displays the fields of the <codeph>STRUCT</codeph> as if
+      they were columns of a table:
+    </p>
+
+<codeblock><![CDATA[DESCRIBE struct_demo.employee_info;
++----------+--------+
+| name     | type   |
++----------+--------+
+| employer | string |
+| id       | bigint |
+| address  | string |
++----------+--------+
+]]>
+</codeblock>
+
+    <p>
+      Because <codeph>PLACES_LIVED</codeph> is a <codeph>STRUCT</codeph> inside an <codeph>ARRAY</codeph>, the initial
+      <codeph>DESCRIBE</codeph> shows the structure of the <codeph>ARRAY</codeph>:
+    </p>
+
+<codeblock><![CDATA[DESCRIBE struct_demo.places_lived;
++------+------------------+
+| name | type             |
++------+------------------+
+| item | struct<          |
+|      |   street:string, |
+|      |   city:string,   |
+|      |   country:string |
+|      | >                |
+| pos  | bigint           |
++------+------------------+
+]]>
+</codeblock>
+
+    <p>
+      Ask for the details of the <codeph>ITEM</codeph> field of the <codeph>ARRAY</codeph> to see just the layout of the
+      <codeph>STRUCT</codeph>:
+    </p>
+
+<codeblock><![CDATA[DESCRIBE struct_demo.places_lived.item;
++---------+--------+
+| name    | type   |
++---------+--------+
+| street  | string |
+| city    | string |
+| country | string |
++---------+--------+
+]]>
+</codeblock>
+
+    <p>
+      Likewise, <codeph>MEMORABLE_MOMENTS</codeph> has a <codeph>STRUCT</codeph> inside a <codeph>MAP</codeph>, which requires an extra
+      level of qualified name to see just the <codeph>STRUCT</codeph> part:
+    </p>
+
+<codeblock><![CDATA[DESCRIBE struct_demo.memorable_moments;
++-------+------------------+
+| name  | type             |
++-------+------------------+
+| key   | string           |
+| value | struct<          |
+|       |   year:int,      |
+|       |   place:string,  |
+|       |   details:string |
+|       | >                |
++-------+------------------+
+]]>
+</codeblock>
+
+    <p>
+      For a <codeph>MAP</codeph>, ask to see the <codeph>VALUE</codeph> field to see the corresponding <codeph>STRUCT</codeph> fields in a
+      table-like structure:
+    </p>
+
+<codeblock><![CDATA[DESCRIBE struct_demo.memorable_moments.value;
++---------+--------+
+| name    | type   |
++---------+--------+
+| year    | int    |
+| place   | string |
+| details | string |
++---------+--------+
+]]>
+</codeblock>
+
+    <p>
+      For a <codeph>STRUCT</codeph> inside a <codeph>STRUCT</codeph>, we can see the fields of the outer <codeph>STRUCT</codeph>:
+    </p>
+
+<codeblock><![CDATA[DESCRIBE struct_demo.current_address;
++----------------+-----------------------+
+| name           | type                  |
++----------------+-----------------------+
+| street_address | struct<               |
+|                |   street_number:int,  |
+|                |   street_name:string, |
+|                |   street_type:string  |
+|                | >                     |
+| country        | string                |
+| postal_code    | string                |
++----------------+-----------------------+
+]]>
+</codeblock>
+
+    <p>
+      Then we can use a further qualified name to see just the fields of the inner <codeph>STRUCT</codeph>:
+    </p>
+
+<codeblock><![CDATA[DESCRIBE struct_demo.current_address.street_address;
++---------------+--------+
+| name          | type   |
++---------------+--------+
+| street_number | int    |
+| street_name   | string |
+| street_type   | string |
++---------------+--------+
+]]>
+</codeblock>
+
+    <p>
+      The following example shows how to examine the structure of a table containing one or more <codeph>STRUCT</codeph> columns by using
+      the <codeph>DESCRIBE</codeph> statement. You can visualize each <codeph>STRUCT</codeph> as its own table, with columns named the same
+      as each field of the <codeph>STRUCT</codeph>. If the <codeph>STRUCT</codeph> is nested inside another complex type, such as
+      <codeph>ARRAY</codeph>, you can extend the qualified name passed to <codeph>DESCRIBE</codeph> until the output shows just the
+      <codeph>STRUCT</codeph> fields.
+    </p>
+
+<!-- To do: See why the most verbose query form gives an error. -->
+
+<codeblock><![CDATA[DESCRIBE struct_demo;
++-------------------+--------------------------+---------+
+| name              | type                     | comment |
++-------------------+--------------------------+---------+
+| id                | bigint                   |         |
+| name              | string                   |         |
+| employee_info     | struct<                  |         |
+|                   |   employer:string,       |         |
+|                   |   id:bigint,             |         |
+|                   |   address:string         |         |
+|                   | >                        |         |
+| places_lived      | array<struct<            |         |
+|                   |   street:string,         |         |
+|                   |   city:string,           |         |
+|                   |   country:string         |         |
+|                   | >>                       |         |
+| memorable_moments | map<string,struct<       |         |
+|                   |   year:int,              |         |
+|                   |   place:string,          |         |
+|                   |   details:string         |         |
+|                   | >>                       |         |
+| current_address   | struct<                  |         |
+|                   |   street_address:struct< |         |
+|                   |     street_number:int,   |         |
+|                   |     street_name:string,  |         |
+|                   |     street_type:string   |         |
+|                   |   >,                     |         |
+|                   |   country:string,        |         |
+|                   |   postal_code:string     |         |
+|                   | >                        |         |
++-------------------+--------------------------+---------+
+
+SELECT id, employee_info.id FROM struct_demo;
+
+SELECT id, employee_info.id AS employee_id FROM struct_demo;
+
+SELECT id, employee_info.id AS employee_id, employee_info.employer
+  FROM struct_demo;
+
+SELECT id, name, street, city, country
+  FROM struct_demo, struct_demo.places_lived;
+
+SELECT id, name, places_lived.pos, places_lived.street, places_lived.city, places_lived.country
+  FROM struct_demo, struct_demo.places_lived;
+
+SELECT id, name, pl.pos, pl.street, pl.city, pl.country
+  FROM struct_demo, struct_demo.places_lived AS pl;
+
+SELECT id, name, places_lived.pos, places_lived.street, places_lived.city, places_lived.country
+  FROM struct_demo, struct_demo.places_lived;
+
+SELECT id, name, pos, street, city, country
+  FROM struct_demo, struct_demo.places_lived;
+
+SELECT id, name, memorable_moments.key,
+  memorable_moments.value.year,
+  memorable_moments.value.place,
+  memorable_moments.value.details
+FROM struct_demo, struct_demo.memorable_moments
+WHERE memorable_moments.key IN ('Birthday','Anniversary','Graduation');
+
+SELECT id, name, mm.key, mm.value.year, mm.value.place, mm.value.details
+  FROM struct_demo, struct_demo.memorable_moments AS mm
+WHERE mm.key IN ('Birthday','Anniversary','Graduation');
+
+SELECT id, name, memorable_moments.key, memorable_moments.value.year,
+  memorable_moments.value.place, memorable_moments.value.details
+FROM struct_demo, struct_demo.memorable_moments
+WHERE key IN ('Birthday','Anniversary','Graduation');
+
+SELECT id, name, key, value.year, value.place, value.details
+  FROM struct_demo, struct_demo.memorable_moments
+WHERE key IN ('Birthday','Anniversary','Graduation');
+
+SELECT id, name, key, year, place, details
+  FROM struct_demo, struct_demo.memorable_moments
+WHERE key IN ('Birthday','Anniversary','Graduation');
+
+SELECT id, name,
+  current_address.street_address.street_number,
+  current_address.street_address.street_name,
+  current_address.street_address.street_type,
+  current_address.country,
+  current_address.postal_code
+FROM struct_demo;
+]]>
+</codeblock>
+
+    <p>
+      For example, this table uses a struct that encodes several data values for each phone number associated with a person. Each person can
+      have a variable-length array of associated phone numbers, and queries can refer to the category field to locate specific home, work,
+      mobile, and so on kinds of phone numbers.
+    </p>
+
+<codeblock><![CDATA[CREATE TABLE contact_info_many_structs
+(
+  id BIGINT, name STRING,
+  phone_numbers ARRAY < STRUCT <category:STRING, country_code:STRING, area_code:SMALLINT, full_number:STRING, mobile:BOOLEAN, carrier:STRING > >
+) STORED AS PARQUET;
+]]>
+</codeblock>
+
+    <p>
+      Because structs are naturally suited to composite values where the fields have different data types, you might use them to decompose
+      things such as addresses:
+    </p>
+
+<codeblock><![CDATA[CREATE TABLE contact_info_detailed_address
+(
+  id BIGINT, name STRING,
+  address STRUCT < house_number:INT, street:STRING, street_type:STRING, apartment:STRING, city:STRING, region:STRING, country:STRING >
+);
+]]>
+</codeblock>
+
+    <p>
+      In a big data context, splitting out data fields such as the number part of the address and the street name could let you do analysis
+      on each field independently. For example, which streets have the largest number range of addresses, what are the statistical
+      properties of the street names, which areas have a higher proportion of <q>Roads</q>, <q>Courts</q> or <q>Boulevards</q>, and so on.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_complex_types.xml#complex_types"/>, <xref href="impala_array.xml#array"/>,
+<!-- <xref href="impala_struct.xml#struct"/>, -->
+      <xref href="impala_map.xml#map"/>
+    </p>
+
+  </conbody>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_subqueries.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_subqueries.xml b/docs/topics/impala_subqueries.xml
new file mode 100644
index 0000000..880f15c
--- /dev/null
+++ b/docs/topics/impala_subqueries.xml
@@ -0,0 +1,320 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="2.0.0" id="subqueries">
+
+  <title>Subqueries in Impala SELECT Statements</title>
+  <titlealts audience="PDF"><navtitle>Subqueries</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.0.0">
+      <indexterm audience="Cloudera">subqueries</indexterm>
+      A <term>subquery</term> is a query that is nested within another query. Subqueries let queries on one table
+      dynamically adapt based on the contents of another table. This technique provides great flexibility and
+      expressive power for SQL queries.
+    </p>
+
+    <p>
+      A subquery can return a result set for use in the <codeph>FROM</codeph> or <codeph>WITH</codeph> clauses, or
+      with operators such as <codeph>IN</codeph> or <codeph>EXISTS</codeph>.
+    </p>
+
+    <p>
+      A <term>scalar subquery</term> produces a result set with a single row containing a single column, typically
+      produced by an aggregation function such as <codeph>MAX()</codeph> or <codeph>SUM()</codeph>. This single
+      result value can be substituted in scalar contexts such as arguments to comparison operators. If the result
+      set is empty, the value of the scalar subquery is <codeph>NULL</codeph>. For example, the following query
+      finds the maximum value of <codeph>T2.Y</codeph> and then substitutes that value into the
+      <codeph>WHERE</codeph> clause of the outer block that queries <codeph>T1</codeph>:
+    </p>
+
+<codeblock>SELECT x FROM t1 WHERE x &gt; (SELECT MAX(y) FROM t2);
+</codeblock>
+
+    <p>
+      <term>Uncorrelated subqueries</term> do not refer to any tables from the outer block of the query. The same
+      value or set of values produced by the subquery is used when evaluating each row from the outer query block.
+      In this example, the subquery returns an arbitrary number of values from <codeph>T2.Y</codeph>, and each
+      value of <codeph>T1.X</codeph> is tested for membership in that same set of values:
+    </p>
+
+<codeblock>SELECT x FROM t1 WHERE x IN (SELECT y FROM t2);
+</codeblock>
+
+    <p>
+      <term>Correlated subqueries</term> compare one or more values from the outer query block to values referenced
+      in the <codeph>WHERE</codeph> clause of the subquery. Each row evaluated by the outer <codeph>WHERE</codeph>
+      clause can be evaluated using a different set of values. These kinds of subqueries are restricted in the
+      kinds of comparisons they can do between columns of the inner and outer tables. (See the following
+      <b>Restrictions</b> item.)
+    </p>
+
+    <p>
+      For example, the following query finds all the employees with salaries that are higher than average for their
+      department. The subquery potentially computes a different <codeph>AVG()</codeph> value for each employee.
+    </p>
+
+<!-- TK: Construct an EMPLOYEES schema to try out examples like these. -->
+
+<codeblock>SELECT employee_name, employee_id FROM employees one WHERE
+  salary &gt; (SELECT avg(salary) FROM employees two WHERE one.dept_id = two.dept_id);
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p>
+      <b>Subquery in the <codeph>FROM</codeph> clause:</b>
+    </p>
+
+<codeblock>SELECT <varname>select_list</varname> FROM <varname>table_ref</varname> [, <varname>table_ref</varname> ...]
+
+<varname>table_ref</varname> ::= <varname>table_name</varname> | (<varname>select_statement</varname>)
+</codeblock>
+
+    <p>
+      <b>Subqueries in <codeph>WHERE</codeph> clause:</b>
+    </p>
+
+<codeblock>WHERE <varname>value</varname> <varname>comparison_operator</varname> (<varname>scalar_select_statement</varname>)
+WHERE <varname>value</varname> [NOT] IN (<varname>select_statement</varname>)
+WHERE [NOT] EXISTS (<varname>correlated_select_statement</varname>)
+WHERE NOT EXISTS (<varname>correlated_select_statement</varname>)
+</codeblock>
+
+    <p>
+      <codeph>comparison_operator</codeph> is a numeric comparison such as <codeph>=</codeph>,
+      <codeph>&lt;=</codeph>, <codeph>!=</codeph>, and so on, or a string comparison operator such as
+      <codeph>LIKE</codeph> or <codeph>REGEXP</codeph>.
+    </p>
+
+    <p rev="2.0.0">
+      Although you can use non-equality comparison operators such as <codeph>&lt;</codeph> or
+      <codeph>&gt;=</codeph>, the subquery must include at least one equality comparison between the columns of the
+      inner and outer query blocks.
+    </p>
+
+    <p>
+      All syntax is available for both correlated and uncorrelated queries, except that the <codeph>NOT
+      EXISTS</codeph> clause cannot be used with an uncorrelated subquery.
+    </p>
+
+    <p>
+      Impala subqueries can be nested arbitrarily deep.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/sql1999"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      This example illustrates how subqueries can be used in the <codeph>FROM</codeph> clause to organize the table
+      names, column names, and column values by producing intermediate result sets, especially for join queries.
+    </p>
+
+<codeblock>SELECT avg(t1.x), max(t2.y) FROM
+  (SELECT id, cast(a AS DECIMAL(10,5)) AS x FROM raw_data WHERE a BETWEEN 0 AND 100) AS t1
+  JOIN
+  (SELECT id, length(s) AS y FROM raw_data WHERE s LIKE 'A%') AS t2;
+  USING (id);
+</codeblock>
+
+    <p rev="2.0.0">
+      These examples show how a query can test for the existence of values in a separate table using the
+      <codeph>EXISTS()</codeph> operator with a subquery.
+<!--
+Internally, these queries are processed in a way similar to join queries.
+Because the values from the second table are not part of the result set, the subquery
+is more efficient than the equivalent join query.
+-->
+    </p>
+
+    <p>
+      The following examples show how a value can be compared against a set of values returned by a subquery.
+    </p>
+
+<codeblock rev="2.0.0">SELECT count(x) FROM t1 WHERE EXISTS(SELECT 1 FROM t2 WHERE t1.x = t2.y * 10);
+
+SELECT x FROM t1 WHERE x IN (SELECT y FROM t2 WHERE state = 'CA');
+</codeblock>
+
+    <p>
+      The following examples demonstrate scalar subqueries. When a subquery is known to return a single value, you
+      can substitute it where you would normally put a constant value.
+    </p>
+
+<codeblock>SELECT x FROM t1 WHERE y = (SELECT max(z) FROM t2);
+SELECT x FROM t1 WHERE y &gt; (SELECT count(z) FROM t2);
+</codeblock>
+
+<!-- <p conref="../shared/impala_common.xml#common/partitioning_blurb"/> -->
+
+<!--
+<p conref="../shared/impala_common.xml#common/hbase_blurb"/>
+<p>
+Currently, the <codeph>IN (<varname>subquery</varname>)</codeph> operator results in a full table scan
+of an HBase table, rather than being translated into a series of single-row lookups.
+Therefore, this is not an efficient construct to use with Impala queries for HBase tables.
+</p>
+-->
+
+<!--
+<p conref="../shared/impala_common.xml#common/parquet_blurb"/>
+<p conref="../shared/impala_common.xml#common/text_blurb"/>
+<p conref="../shared/impala_common.xml#common/compatibility_blurb"/>
+-->
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      If the same table is referenced in both the outer and inner query blocks, construct a table alias in the
+      outer query block and use a fully qualified name to distinguish the inner and outer table references:
+    </p>
+
+<!-- TK: verify the logic of this example. Probably have other similar ones that could be reused here. -->
+
+<codeblock>SELECT * FROM t1 one WHERE id IN (SELECT parent FROM t1 two WHERE t1.parent = t2.id);
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/internals_blurb"/>
+
+    <p>
+      Internally, subqueries involving <codeph>IN</codeph>, <codeph>NOT IN</codeph>, <codeph>EXISTS</codeph>, or
+      <codeph>NOT EXISTS</codeph> clauses are rewritten into join queries. Depending on the syntax, the subquery
+      might be rewritten to an outer join, semi join, cross join, or anti join.
+    </p>
+
+    <p>
+      A query is processed differently depending on whether the subquery calls any aggregation functions. There are
+      correlated and uncorrelated forms, with and without calls to aggregation functions. Each of these four
+      categories is rewritten differently.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/column_stats_blurb"/>
+
+    <p>
+      Because queries that include correlated and uncorrelated subqueries in the <codeph>WHERE</codeph> clause are
+      written into join queries, to achieve best performance, follow the same guidelines for running the
+      <codeph>COMPUTE STATS</codeph> statement as you do for tables involved in regular join queries. Run the
+      <codeph>COMPUTE STATS</codeph> statement for each associated tables after loading or substantially changing
+      the data in that table. See <xref href="impala_perf_stats.xml#perf_stats"/> for details.
+    </p>
+
+    <p>
+      <b>Added in:</b> Subqueries are substantially enhanced starting in Impala 2.0 for CDH 4, and CDH 5.2.0. Now,
+      they can be used in the <codeph>WHERE</codeph> clause, in combination with clauses such as
+      <codeph>EXISTS</codeph> and <codeph>IN</codeph>, rather than just in the <codeph>FROM</codeph> clause.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+    <p>
+      The initial Impala support for nested subqueries addresses the most common use cases. Some restrictions
+      remain:
+    </p>
+
+    <ul>
+      <li>
+        <p>
+          Although you can use subqueries in a query involving <codeph>UNION</codeph> or <codeph>UNION ALL</codeph>
+          in Impala 2.1.0 and higher, currently you cannot construct a union of two subqueries (for example, in the
+          argument of an <codeph>IN</codeph> or <codeph>EXISTS</codeph> operator).
+        </p>
+      </li>
+
+      <li>
+        <p>
+          Subqueries returning scalar values cannot be used with the operators <codeph>ANY</codeph> or
+          <codeph>ALL</codeph>. (Impala does not currently have a <codeph>SOME</codeph> operator, but if it did,
+          the same restriction would apply.)
+        </p>
+      </li>
+
+      <li>
+        <p>
+          For the <codeph>EXISTS</codeph> and <codeph>NOT EXISTS</codeph> clauses, any subquery comparing values
+          from the outer query block to another table must use at least one equality comparison, not exclusively
+          other kinds of comparisons such as less than, greater than, <codeph>BETWEEN</codeph>, or
+          <codeph>!=</codeph>.
+        </p>
+      </li>
+
+      <li>
+<!-- TK: think this is no longer true. -->
+        <p>
+          Currently, a scalar subquery cannot be used as the first or second argument to the
+          <codeph>BETWEEN</codeph> operator.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          A subquery cannot be used inside an <codeph>OR</codeph> conjunction. Expressions inside a subquery, for
+          example in the <codeph>WHERE</codeph> clause, can use <codeph>OR</codeph> conjunctions; the restriction
+          only applies to parts of the query <q>above</q> the subquery.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          Scalar subqueries are only supported in numeric contexts. You cannot use a scalar subquery as an argument
+          to the <codeph>LIKE</codeph>, <codeph>REGEXP</codeph>, or <codeph>RLIKE</codeph> operators, or compare it
+          to a value of a non-numeric type such as <codeph>TIMESTAMP</codeph> or <codeph>BOOLEAN</codeph>.
+        </p>
+      </li>
+
+      <li>
+        <p>
+<!-- A subquery cannot be used to generate a scalar value for a function call. -->
+          You cannot use subqueries with the <codeph>CASE</codeph> function to generate the comparison value, the
+          values to be compared against, or the return value.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          A subquery is not allowed in the filter condition for the <codeph>HAVING</codeph> clause. (Strictly
+          speaking, a subquery cannot appear anywhere outside the <codeph>WITH</codeph>, <codeph>FROM</codeph>, and
+          <codeph>WHERE</codeph> clauses.)
+        </p>
+      </li>
+
+      <li>
+        <p>
+          You must use a fully qualified name
+          (<codeph><varname>table_name</varname>.<varname>column_name</varname></codeph> or
+          <codeph><varname>database_name</varname>.<varname>table_name</varname>.<varname>column_name</varname></codeph>)
+          when referring to any column from the outer query block within a subquery.
+        </p>
+      </li>
+    </ul>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+    <p rev="2.3.0">
+      For the complex types (<codeph>ARRAY</codeph>, <codeph>STRUCT</codeph>, and <codeph>MAP</codeph>)
+      available in <keyword keyref="impala23_full"/> and higher, the join queries that <q>unpack</q> complex type
+      columns often use correlated subqueries in the <codeph>FROM</codeph> clause.
+      For example, if the first table in the join clause is <codeph>CUSTOMER</codeph>, the second
+      join clause might have a subquery that selects from the column <codeph>CUSTOMER.C_ORDERS</codeph>,
+      which is an <codeph>ARRAY</codeph>. The subquery re-evaluates the <codeph>ARRAY</codeph> elements
+      corresponding to each row from the <codeph>CUSTOMER</codeph> table.
+      See <xref href="impala_complex_types.xml#complex_types"/> for details and examples of
+      using subqueries with complex types.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_operators.xml#exists"/>, <xref href="impala_operators.xml#in"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_sum.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_sum.xml b/docs/topics/impala_sum.xml
new file mode 100644
index 0000000..cb9c453
--- /dev/null
+++ b/docs/topics/impala_sum.xml
@@ -0,0 +1,239 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="sum">
+
+  <title>SUM Function</title>
+  <titlealts audience="PDF"><navtitle>SUM</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="Analytic Functions"/>
+      <data name="Category" value="Aggregate Functions"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">sum() function</indexterm>
+      An aggregate function that returns the sum of a set of numbers. Its single argument can be numeric column, or
+      the numeric result of a function or expression applied to the column value. Rows with a <codeph>NULL</codeph>
+      value for the specified column are ignored. If the table is empty, or all the values supplied to
+      <codeph>MIN</codeph> are <codeph>NULL</codeph>, <codeph>SUM</codeph> returns <codeph>NULL</codeph>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>SUM([DISTINCT | ALL] <varname>expression</varname>) [OVER (<varname>analytic_clause</varname>)]</codeblock>
+
+    <p>
+      When the query contains a <codeph>GROUP BY</codeph> clause, returns one value for each combination of
+      grouping values.
+    </p>
+
+    <p>
+      <b>Return type:</b> <codeph>BIGINT</codeph> for integer arguments, <codeph>DOUBLE</codeph> for floating-point
+      arguments
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_aggregation_explanation"/>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_aggregation_example"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      The following example shows how to use <codeph>SUM()</codeph> to compute the total for all the values in the
+      table, a subset of values, or the sum for each combination of values in the <codeph>GROUP BY</codeph> clause:
+    </p>
+
+<codeblock>-- Total all the values for this column in the table.
+select sum(c1) from t1;
+-- Find the total for this column from a subset of the table.
+select sum(c1) from t1 where month = 'January' and year = '2013';
+-- Find the total from a set of numeric function results.
+select sum(length(s)) from t1;
+-- Often used with functions that return predefined values to compute a score.
+select sum(case when grade = 'A' then 1.0 when grade = 'B' then 0.75 else 0) as class_honors from test_scores;
+-- Can also be used in combination with DISTINCT and/or GROUP BY.
+-- Return more than one result.
+select month, year, sum(purchase_price) from store_stats group by month, year;
+-- Filter the input to eliminate duplicates before performing the calculation.
+select sum(distinct x) from t1;
+</codeblock>
+
+    <p rev="2.0.0">
+      The following examples show how to use <codeph>SUM()</codeph> in an analytic context. They use a table
+      containing integers from 1 to 10. Notice how the <codeph>SUM()</codeph> is reported for each input value, as
+      opposed to the <codeph>GROUP BY</codeph> clause which condenses the result set.
+<codeblock>select x, property, sum(x) <b>over (partition by property)</b> as sum from int_t where property in ('odd','even');
++----+----------+-----+
+| x  | property | sum |
++----+----------+-----+
+| 2  | even     | 30  |
+| 4  | even     | 30  |
+| 6  | even     | 30  |
+| 8  | even     | 30  |
+| 10 | even     | 30  |
+| 1  | odd      | 25  |
+| 3  | odd      | 25  |
+| 5  | odd      | 25  |
+| 7  | odd      | 25  |
+| 9  | odd      | 25  |
++----+----------+-----+
+</codeblock>
+
+Adding an <codeph>ORDER BY</codeph> clause lets you experiment with results that are cumulative or apply to a moving
+set of rows (the <q>window</q>). The following examples use <codeph>SUM()</codeph> in an analytic context
+(that is, with an <codeph>OVER()</codeph> clause) to produce a running total of all the even values,
+then a running total of all the odd values. The basic <codeph>ORDER BY x</codeph> clause implicitly
+activates a window clause of <codeph>RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW</codeph>,
+which is effectively the same as <codeph>ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW</codeph>,
+therefore all of these examples produce the same results:
+<codeblock>select x, property,
+  sum(x) over (partition by property <b>order by x</b>) as 'cumulative total'
+  from int_t where property in ('odd','even');
++----+----------+------------------+
+| x  | property | cumulative total |
++----+----------+------------------+
+| 2  | even     | 2                |
+| 4  | even     | 6                |
+| 6  | even     | 12               |
+| 8  | even     | 20               |
+| 10 | even     | 30               |
+| 1  | odd      | 1                |
+| 3  | odd      | 4                |
+| 5  | odd      | 9                |
+| 7  | odd      | 16               |
+| 9  | odd      | 25               |
++----+----------+------------------+
+
+select x, property,
+  sum(x) over
+  (
+    partition by property
+    <b>order by x</b>
+    <b>range between unbounded preceding and current row</b>
+  ) as 'cumulative total'
+from int_t where property in ('odd','even');
++----+----------+------------------+
+| x  | property | cumulative total |
++----+----------+------------------+
+| 2  | even     | 2                |
+| 4  | even     | 6                |
+| 6  | even     | 12               |
+| 8  | even     | 20               |
+| 10 | even     | 30               |
+| 1  | odd      | 1                |
+| 3  | odd      | 4                |
+| 5  | odd      | 9                |
+| 7  | odd      | 16               |
+| 9  | odd      | 25               |
++----+----------+------------------+
+
+select x, property,
+  sum(x) over
+  (
+    partition by property
+    <b>order by x</b>
+    <b>rows between unbounded preceding and current row</b>
+  ) as 'cumulative total'
+  from int_t where property in ('odd','even');
++----+----------+------------------+
+| x  | property | cumulative total |
++----+----------+------------------+
+| 2  | even     | 2                |
+| 4  | even     | 6                |
+| 6  | even     | 12               |
+| 8  | even     | 20               |
+| 10 | even     | 30               |
+| 1  | odd      | 1                |
+| 3  | odd      | 4                |
+| 5  | odd      | 9                |
+| 7  | odd      | 16               |
+| 9  | odd      | 25               |
++----+----------+------------------+
+</codeblock>
+
+Changing the direction of the <codeph>ORDER BY</codeph> clause causes the intermediate
+results of the cumulative total to be calculated in a different order:
+
+<codeblock>select sum(x) over (partition by property <b>order by x desc</b>) as 'cumulative total'
+  from int_t where property in ('odd','even');
++----+----------+------------------+
+| x  | property | cumulative total |
++----+----------+------------------+
+| 10 | even     | 10               |
+| 8  | even     | 18               |
+| 6  | even     | 24               |
+| 4  | even     | 28               |
+| 2  | even     | 30               |
+| 9  | odd      | 9                |
+| 7  | odd      | 16               |
+| 5  | odd      | 21               |
+| 3  | odd      | 24               |
+| 1  | odd      | 25               |
++----+----------+------------------+
+</codeblock>
+
+The following examples show how to construct a moving window, with a running total taking into account 1 row before
+and 1 row after the current row, within the same partition (all the even values or all the odd values).
+Because of a restriction in the Impala <codeph>RANGE</codeph> syntax, this type of
+moving window is possible with the <codeph>ROWS BETWEEN</codeph> clause but not the <codeph>RANGE BETWEEN</codeph>
+clause:
+<codeblock>select x, property,
+  sum(x) over
+  (
+    partition by property
+    <b>order by x</b>
+    <b>rows between 1 preceding and 1 following</b>
+  ) as 'moving total'
+  from int_t where property in ('odd','even');
++----+----------+--------------+
+| x  | property | moving total |
++----+----------+--------------+
+| 2  | even     | 6            |
+| 4  | even     | 12           |
+| 6  | even     | 18           |
+| 8  | even     | 24           |
+| 10 | even     | 18           |
+| 1  | odd      | 4            |
+| 3  | odd      | 9            |
+| 5  | odd      | 15           |
+| 7  | odd      | 21           |
+| 9  | odd      | 16           |
++----+----------+--------------+
+
+-- Doesn't work because of syntax restriction on RANGE clause.
+select x, property,
+  sum(x) over
+  (
+    partition by property
+    <b>order by x</b>
+    <b>range between 1 preceding and 1 following</b>
+  ) as 'moving total'
+from int_t where property in ('odd','even');
+ERROR: AnalysisException: RANGE is only supported with both the lower and upper bounds UNBOUNDED or one UNBOUNDED and the other CURRENT ROW.
+</codeblock>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+<!-- This conref appears under SUM(), AVG(), FLOAT, and DOUBLE topics. -->
+
+    <p conref="../shared/impala_common.xml#common/sum_double"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_analytic_functions.xml#analytic_functions"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_support_start_over.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_support_start_over.xml b/docs/topics/impala_support_start_over.xml
new file mode 100644
index 0000000..906efaa
--- /dev/null
+++ b/docs/topics/impala_support_start_over.xml
@@ -0,0 +1,30 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="support_start_over">
+
+  <title>SUPPORT_START_OVER Query Option</title>
+  <titlealts audience="PDF"><navtitle>SUPPORT_START_OVER</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">SUPPORT_START_OVER query option</indexterm>
+      Leave this setting at its default value.
+      It is a read-only setting, tested by some client applications such as Hue.
+    </p>
+    <p>
+      If you accidentally change it through <cmdname>impala-shell</cmdname>,
+      subsequent queries encounter errors until you undo the change
+      by issuing <codeph>UNSET support_start_over</codeph>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_boolean"/>
+    <p conref="../shared/impala_common.xml#common/default_false"/>
+  </conbody>
+</concept>



[41/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_char.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_char.xml b/docs/topics/impala_char.xml
new file mode 100644
index 0000000..0298d57
--- /dev/null
+++ b/docs/topics/impala_char.xml
@@ -0,0 +1,278 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="char" rev="2.0.0">
+
+  <title>CHAR Data Type (<keyword keyref="impala20"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>CHAR</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.0.0">
+      <indexterm audience="Cloudera">CHAR data type</indexterm>
+      A fixed-length character type, padded with trailing spaces if necessary to achieve the specified length. If
+      values are longer than the specified length, Impala truncates any trailing characters.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p>
+      In the column definition of a <codeph>CREATE TABLE</codeph> statement:
+    </p>
+
+<codeblock><varname>column_name</varname> CHAR(<varname>length</varname>)</codeblock>
+
+    <p>
+      The maximum length you can specify is 255.
+    </p>
+
+    <p>
+      <b>Semantics of trailing spaces:</b>
+    </p>
+
+    <ul>
+      <li>
+        When you store a <codeph>CHAR</codeph> value shorter than the specified length in a table, queries return
+        the value padded with trailing spaces if necessary; the resulting value has the same length as specified in
+        the column definition.
+      </li>
+
+      <li>
+        If you store a <codeph>CHAR</codeph> value containing trailing spaces in a table, those trailing spaces are
+        not stored in the data file. When the value is retrieved by a query, the result could have a different
+        number of trailing spaces. That is, the value includes however many spaces are needed to pad it to the
+        specified length of the column.
+      </li>
+
+      <li>
+        If you compare two <codeph>CHAR</codeph> values that differ only in the number of trailing spaces, those
+        values are considered identical.
+      </li>
+    </ul>
+
+    <p conref="../shared/impala_common.xml#common/partitioning_bad"/>
+
+    <p conref="../shared/impala_common.xml#common/hbase_no"/>
+
+    <p conref="../shared/impala_common.xml#common/parquet_blurb"/>
+
+    <ul>
+      <li>
+        This type can be read from and written to Parquet files.
+      </li>
+
+      <li>
+        There is no requirement for a particular level of Parquet.
+      </li>
+
+      <li>
+        Parquet files generated by Impala and containing this type can be freely interchanged with other components
+        such as Hive and MapReduce.
+      </li>
+
+      <li>
+        Any trailing spaces, whether implicitly or explicitly specified, are not written to the Parquet data files.
+      </li>
+
+      <li>
+        Parquet data files might contain values that are longer than allowed by the
+        <codeph>CHAR(<varname>n</varname>)</codeph> length limit. Impala ignores any extra trailing characters when
+        it processes those values during a query.
+      </li>
+    </ul>
+
+    <p conref="../shared/impala_common.xml#common/text_blurb"/>
+
+    <p>
+      Text data files might contain values that are longer than allowed for a particular
+      <codeph>CHAR(<varname>n</varname>)</codeph> column. Any extra trailing characters are ignored when Impala
+      processes those values during a query. Text data files can also contain values that are shorter than the
+      defined length limit, and Impala pads them with trailing spaces up to the specified length. Any text data
+      files produced by Impala <codeph>INSERT</codeph> statements do not include any trailing blanks for
+      <codeph>CHAR</codeph> columns.
+    </p>
+
+    <p><b>Avro considerations:</b></p>
+    <p conref="../shared/impala_common.xml#common/avro_2gb_strings"/>
+
+    <p conref="../shared/impala_common.xml#common/compatibility_blurb"/>
+
+    <p>
+      This type is available using Impala 2.0 or higher under CDH 4, or with Impala on CDH 5.2 or higher. There are
+      no compatibility issues with other components when exchanging data files or running Impala on CDH 4.
+    </p>
+
+    <p>
+      Some other database systems make the length specification optional. For Impala, the length is required.
+    </p>
+
+<!--
+<p>
+The Impala maximum length is larger than for the <codeph>CHAR</codeph> data type in Hive.
+If a Hive query encounters a <codeph>CHAR</codeph> value longer than 255 during processing,
+it silently treats the value as length 255.
+</p>
+-->
+
+    <p conref="../shared/impala_common.xml#common/internals_max_bytes"/>
+
+    <p conref="../shared/impala_common.xml#common/added_in_20"/>
+
+    <p conref="../shared/impala_common.xml#common/column_stats_constant"/>
+
+<!-- Seems like a logical design decision but don't think it's currently implemented like this.
+<p>
+Because both the maximum and average length are always known and always the same for
+any given <codeph>CHAR(<varname>n</varname>)</codeph> column, those fields are always filled
+in for <codeph>SHOW COLUMN STATS</codeph> output, even before you run
+<codeph>COMPUTE STATS</codeph> on the table.
+</p>
+-->
+
+    <p conref="../shared/impala_common.xml#common/udf_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      These examples show how trailing spaces are not considered significant when comparing or processing
+      <codeph>CHAR</codeph> values. <codeph>CAST()</codeph> truncates any longer string to fit within the defined
+      length. If a <codeph>CHAR</codeph> value is shorter than the specified length, it is padded on the right with
+      spaces until it matches the specified length. Therefore, <codeph>LENGTH()</codeph> represents the length
+      including any trailing spaces, and <codeph>CONCAT()</codeph> also treats the column value as if it has
+      trailing spaces.
+    </p>
+
+<codeblock>select cast('x' as char(4)) = cast('x   ' as char(4)) as "unpadded equal to padded";
++--------------------------+
+| unpadded equal to padded |
++--------------------------+
+| true                     |
++--------------------------+
+
+create table char_length(c char(3));
+insert into char_length values (cast('1' as char(3))), (cast('12' as char(3))), (cast('123' as char(3))), (cast('123456' as char(3)));
+select concat("[",c,"]") as c, length(c) from char_length;
++-------+-----------+
+| c     | length(c) |
++-------+-----------+
+| [1  ] | 3         |
+| [12 ] | 3         |
+| [123] | 3         |
+| [123] | 3         |
++-------+-----------+
+</codeblock>
+
+    <p>
+      This example shows a case where data values are known to have a specific length, where <codeph>CHAR</codeph>
+      is a logical data type to use.
+<!--
+Because all the <codeph>CHAR</codeph> values have a constant predictable length,
+Impala can efficiently analyze how best to use these values in join queries,
+aggregation queries, and other contexts where column length is significant.
+-->
+    </p>
+
+<codeblock>create table addresses
+  (id bigint,
+   street_name string,
+   state_abbreviation char(2),
+   country_abbreviation char(2));
+</codeblock>
+
+    <p>
+      The following example shows how values written by Impala do not physically include the trailing spaces. It
+      creates a table using text format, with <codeph>CHAR</codeph> values much shorter than the declared length,
+      and then prints the resulting data file to show that the delimited values are not separated by spaces. The
+      same behavior applies to binary-format Parquet data files.
+    </p>
+
+<codeblock>create table char_in_text (a char(20), b char(30), c char(40))
+  row format delimited fields terminated by ',';
+
+insert into char_in_text values (cast('foo' as char(20)), cast('bar' as char(30)), cast('baz' as char(40))), (cast('hello' as char(20)), cast('goodbye' as char(30)), cast('aloha' as char(40)));
+
+-- Running this Linux command inside impala-shell using the ! shortcut.
+!hdfs dfs -cat 'hdfs://127.0.0.1:8020/user/hive/warehouse/impala_doc_testing.db/char_in_text/*.*';
+foo,bar,baz
+hello,goodbye,aloha
+</codeblock>
+
+    <p>
+      The following example further illustrates the treatment of spaces. It replaces the contents of the previous
+      table with some values including leading spaces, trailing spaces, or both. Any leading spaces are preserved
+      within the data file, but trailing spaces are discarded. Then when the values are retrieved by a query, the
+      leading spaces are retrieved verbatim while any necessary trailing spaces are supplied by Impala.
+    </p>
+
+<codeblock>insert overwrite char_in_text values (cast('trailing   ' as char(20)), cast('   leading and trailing   ' as char(30)), cast('   leading' as char(40)));
+!hdfs dfs -cat 'hdfs://127.0.0.1:8020/user/hive/warehouse/impala_doc_testing.db/char_in_text/*.*';
+trailing,   leading and trailing,   leading
+
+select concat('[',a,']') as a, concat('[',b,']') as b, concat('[',c,']') as c from char_in_text;
++------------------------+----------------------------------+--------------------------------------------+
+| a                      | b                                | c                                          |
++------------------------+----------------------------------+--------------------------------------------+
+| [trailing            ] | [   leading and trailing       ] | [   leading                              ] |
++------------------------+----------------------------------+--------------------------------------------+
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+    <p>
+      Because the blank-padding behavior requires allocating the maximum length for each value in memory, for
+      scalability reasons avoid declaring <codeph>CHAR</codeph> columns that are much longer than typical values in
+      that column.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/blobs_are_strings"/>
+
+    <p>
+      When an expression compares a <codeph>CHAR</codeph> with a <codeph>STRING</codeph> or
+      <codeph>VARCHAR</codeph>, the <codeph>CHAR</codeph> value is implicitly converted to <codeph>STRING</codeph>
+      first, with trailing spaces preserved.
+    </p>
+
+<codeblock>select cast("foo  " as char(5)) = 'foo' as "char equal to string";
++----------------------+
+| char equal to string |
++----------------------+
+| false                |
++----------------------+
+</codeblock>
+
+    <p>
+      This behavior differs from other popular database systems. To get the expected result of
+      <codeph>TRUE</codeph>, cast the expressions on both sides to <codeph>CHAR</codeph> values of the appropriate
+      length:
+    </p>
+
+<codeblock>select cast("foo  " as char(5)) = cast('foo' as char(3)) as "char equal to string";
++----------------------+
+| char equal to string |
++----------------------+
+| true                 |
++----------------------+
+</codeblock>
+
+    <p>
+      This behavior is subject to change in future releases.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_string.xml#string"/>, <xref href="impala_varchar.xml#varchar"/>,
+      <xref href="impala_literals.xml#string_literals"/>,
+      <xref href="impala_string_functions.xml#string_functions"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_cluster_sizing.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_cluster_sizing.xml b/docs/topics/impala_cluster_sizing.xml
new file mode 100644
index 0000000..382f68c
--- /dev/null
+++ b/docs/topics/impala_cluster_sizing.xml
@@ -0,0 +1,353 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="cluster_sizing">
+
+  <title>Cluster Sizing Guidelines for Impala</title>
+  <titlealts audience="PDF"><navtitle>Cluster Sizing</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Clusters"/>
+      <data name="Category" value="Planning"/>
+      <data name="Category" value="Sizing"/>
+      <data name="Category" value="Deploying"/>
+      <!-- Hoist by my own petard. Memory is an important theme of this topic but that's in a <section> title. -->
+      <data name="Category" value="Sectionated Pages"/>
+      <data name="Category" value="Memory"/>
+      <data name="Category" value="Scalability"/>
+      <data name="Category" value="Proof of Concept"/>
+      <data name="Category" value="Requirements"/>
+      <data name="Category" value="Guidelines"/>
+      <data name="Category" value="Best Practices"/>
+      <data name="Category" value="Administrators"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">cluster sizing</indexterm>
+      This document provides a very rough guideline to estimate the size of a cluster needed for a specific
+      customer application. You can use this information when planning how much and what type of hardware to
+      acquire for a new cluster, or when adding Impala workloads to an existing cluster.
+    </p>
+
+    <note>
+      Before making purchase or deployment decisions, consult your Cloudera representative to verify the
+      conclusions about hardware requirements based on your data volume and workload.
+    </note>
+
+<!--    <p outputclass="toc inpage"/> -->
+
+    <p>
+      Always use hosts with identical specifications and capacities for all the nodes in the cluster. Currently,
+      Impala divides the work evenly between cluster nodes, regardless of their exact hardware configuration.
+      Because work can be distributed in different ways for different queries, if some hosts are overloaded
+      compared to others in terms of CPU, memory, I/O, or network, you might experience inconsistent performance
+      and overall slowness
+    </p>
+
+    <p>
+      For analytic workloads with star/snowflake schemas, and using consistent hardware for all nodes (64 GB RAM,
+      12 2 TB hard drives, 2x E5-2630L 12 cores total, 10 GB network), the following table estimates the number of
+      DataNodes needed in the cluster based on data size and the number of concurrent queries, for workloads
+      similar to TPC-DS benchmark queries:
+    </p>
+
+    <table>
+      <title>Cluster size estimation based on the number of concurrent queries and data size with a 20 second average query response time</title>
+      <tgroup cols="6">
+        <colspec colnum="1" colname="col1"/>
+        <colspec colnum="2" colname="col2"/>
+        <colspec colnum="3" colname="col3"/>
+        <colspec colnum="4" colname="col4"/>
+        <colspec colnum="5" colname="col5"/>
+        <colspec colnum="6" colname="col6"/>
+        <thead>
+          <row>
+            <entry>
+              Data Size
+            </entry>
+            <entry>
+              1 query
+            </entry>
+            <entry>
+              10 queries
+            </entry>
+            <entry>
+              100 queries
+            </entry>
+            <entry>
+              1000 queries
+            </entry>
+            <entry>
+              2000 queries
+            </entry>
+          </row>
+        </thead>
+        <tbody>
+          <row>
+            <entry>
+              <b>250 GB</b>
+            </entry>
+            <entry>
+              2
+            </entry>
+            <entry>
+              2
+            </entry>
+            <entry>
+              5
+            </entry>
+            <entry>
+              35
+            </entry>
+            <entry>
+              70
+            </entry>
+          </row>
+          <row>
+            <entry>
+              <b>500 GB</b>
+            </entry>
+            <entry>
+              2
+            </entry>
+            <entry>
+              2
+            </entry>
+            <entry>
+              10
+            </entry>
+            <entry>
+              70
+            </entry>
+            <entry>
+              135
+            </entry>
+          </row>
+          <row>
+            <entry>
+              <b>1 TB</b>
+            </entry>
+            <entry>
+              2
+            </entry>
+            <entry>
+              2
+            </entry>
+            <entry>
+              15
+            </entry>
+            <entry>
+              135
+            </entry>
+            <entry>
+              270
+            </entry>
+          </row>
+          <row>
+            <entry>
+              <b>15 TB</b>
+            </entry>
+            <entry>
+              2
+            </entry>
+            <entry>
+              20
+            </entry>
+            <entry>
+              200
+            </entry>
+            <entry>
+              N/A
+            </entry>
+            <entry>
+              N/A
+            </entry>
+          </row>
+          <row>
+            <entry>
+              <b>30 TB</b>
+            </entry>
+            <entry>
+              4
+            </entry>
+            <entry>
+              40
+            </entry>
+            <entry>
+              400
+            </entry>
+            <entry>
+              N/A
+            </entry>
+            <entry>
+              N/A
+            </entry>
+          </row>
+          <row>
+            <entry>
+              <b>60 TB</b>
+            </entry>
+            <entry>
+              8
+            </entry>
+            <entry>
+              80
+            </entry>
+            <entry>
+              800
+            </entry>
+            <entry>
+              N/A
+            </entry>
+            <entry>
+              N/A
+            </entry>
+          </row>
+        </tbody>
+      </tgroup>
+    </table>
+
+    <section id="sizing_factors">
+
+      <title>Factors Affecting Scalability</title>
+
+      <p>
+        A typical analytic workload (TPC-DS style queries) using recommended hardware is usually CPU-bound. Each
+        node can process roughly 1.6 GB/sec. Both CPU-bound and disk-bound workloads can scale almost linearly with
+        cluster size. However, for some workloads, the scalability might be bounded by the network, or even by
+        memory.
+      </p>
+
+      <p>
+        If the workload is already network bound (on a 10 GB network), increasing the cluster size won\u2019t reduce
+        the network load; in fact, a larger cluster could increase network traffic because some queries involve
+        <q>broadcast</q> operations to all DataNodes. Therefore, boosting the cluster size does not improve query
+        throughput in a network-constrained environment.
+      </p>
+
+      <p>
+        Let\u2019s look at a memory-bound workload. A workload is memory-bound if Impala cannot run any additional
+        concurrent queries because all memory allocated has already been consumed, but neither CPU, disk, nor
+        network is saturated yet. This can happen because currently Impala uses only a single core per node to
+        process join and aggregation queries. For a node with 128 GB of RAM, if a join node takes 50 GB, the system
+        cannot run more than 2 such queries at the same time.
+      </p>
+
+      <p>
+        Therefore, at most 2 cores are used. Throughput can still scale almost linearly even for a memory-bound
+        workload. It\u2019s just that the CPU will not be saturated. Per-node throughput will be lower than 1.6
+        GB/sec. Consider increasing the memory per node.
+      </p>
+
+      <p>
+        As long as the workload is not network- or memory-bound, we can use the 1.6 GB/second per node as the
+        throughput estimate.
+      </p>
+    </section>
+
+    <section id="sizing_details">
+
+      <title>A More Precise Approach</title>
+
+      <p>
+        A more precise sizing estimate would require not only queries per minute (QPM), but also an average data
+        size scanned per query (D). With the proper partitioning strategy, D is usually a fraction of the total
+        data size. The following equation can be used as a rough guide to estimate the number of nodes (N) needed:
+      </p>
+
+<codeblock>Eq 1: N &gt; QPM * D / 100 GB
+</codeblock>
+
+      <p>
+        Here is an example. Suppose, on average, a query scans 50 GB of data and the average response time is
+        required to be 15 seconds or less when there are 100 concurrent queries. The QPM is 100/15*60 = 400. We can
+        estimate the number of node using our equation above.
+      </p>
+
+<codeblock>N &gt; QPM * D / 100GB
+N &gt; 400 * 50GB / 100GB
+N &gt; 200
+</codeblock>
+
+      <p>
+        Because this figure is a rough estimate, the corresponding number of nodes could be between 100 and 500.
+      </p>
+
+      <p>
+        Depending on the complexity of the query, the processing rate of query might change. If the query has more
+        joins, aggregation functions, or CPU-intensive functions such as string processing or complex UDFs, the
+        process rate will be lower than 1.6 GB/second per node. On the other hand, if the query only does scan and
+        filtering on numbers, the processing rate can be higher.
+      </p>
+    </section>
+
+    <section id="sizing_mem_estimate">
+
+      <title>Estimating Memory Requirements</title>
+      <!--
+  <prolog>
+    <metadata>
+      <data name="Category" value="Memory"/>
+    </metadata>
+  </prolog>
+      -->
+
+      <p>
+        Impala can handle joins between multiple large tables. Make sure that statistics are collected for all the
+        joined tables, using the <codeph><xref href="impala_compute_stats.xml#compute_stats">COMPUTE
+        STATS</xref></codeph> statement. However, joining big tables does consume more memory. Follow the steps
+        below to calculate the minimum memory requirement.
+      </p>
+
+      <p>
+        Suppose you are running the following join:
+      </p>
+
+<codeblock>select a.*, b.col_1, b.col_2, \u2026 b.col_n
+from a, b
+where a.key = b.key
+and b.col_1 in (1,2,4...)
+and b.col_4 in (....);
+</codeblock>
+
+      <p>
+        And suppose table <codeph>B</codeph> is smaller than table <codeph>A</codeph> (but still a large table).
+      </p>
+
+      <p>
+        The memory requirement for the query is the right-hand table (<codeph>B</codeph>), after decompression,
+        filtering (<codeph>b.col_n in ...</codeph>) and after projection (only using certain columns) must be less
+        than the total memory of the entire cluster.
+      </p>
+
+<codeblock>Cluster Total Memory Requirement  = Size of the smaller table *
+  selectivity factor from the predicate *
+  projection factor * compression ratio
+</codeblock>
+
+      <p>
+        In this case, assume that table <codeph>B</codeph> is 100 TB in Parquet format with 200 columns. The
+        predicate on <codeph>B</codeph> (<codeph>b.col_1 in ...and b.col_4 in ...</codeph>) will select only 10% of
+        the rows from <codeph>B</codeph> and for projection, we are only projecting 5 columns out of 200 columns.
+        Usually, Snappy compression gives us 3 times compression, so we estimate a 3x compression factor.
+      </p>
+
+<codeblock>Cluster Total Memory Requirement  = Size of the smaller table *
+  selectivity factor from the predicate *
+  projection factor * compression ratio
+  = 100TB * 10% * 5/200 * 3
+  = 0.75TB
+  = 750GB
+</codeblock>
+
+      <p>
+        So, if you have a 10-node cluster, each node has 128 GB of RAM and you give 80% to Impala, then you have 1
+        TB of usable memory for Impala, which is more than 750GB. Therefore, your cluster can handle join queries
+        of this magnitude.
+      </p>
+    </section>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_cm_installation.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_cm_installation.xml b/docs/topics/impala_cm_installation.xml
new file mode 100644
index 0000000..2cc2ac5
--- /dev/null
+++ b/docs/topics/impala_cm_installation.xml
@@ -0,0 +1,56 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="cm_installation">
+
+  <title>Installing Impala with Cloudera Manager</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Installing"/>
+      <data name="Category" value="Cloudera Manager"/>
+      <data name="Category" value="Administrators"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Before installing Impala through the Cloudera Manager interface, make sure all applicable nodes have the
+      appropriate hardware configuration and levels of operating system and CDH. See
+      <xref href="impala_prereqs.xml#prereqs"/> for details.
+    </p>
+
+    <note rev="1.2.0">
+      <p rev="1.2.0">
+        To install the latest Impala under CDH 4, upgrade Cloudera Manager to 4.8 or higher. Cloudera Manager 4.8 is
+        the first release that can manage the Impala catalog service introduced in Impala 1.2. Cloudera Manager 4.8
+        requires this service to be present, so if you upgrade to Cloudera Manager 4.8, also upgrade Impala to the
+        most recent version at the same time.
+<!-- Not so relevant now for 1.1.1, but maybe someday we'll capture all this history in a compatibility grid.
+        Upgrade to Cloudera Manager 4.6.2 or higher to enable Cloudera Manager to
+        handle access control for the Impala web UI, available by default through
+        port 25000 on each Impala host.
+        -->
+      </p>
+    </note>
+
+    <p>
+      For information on installing Impala in a Cloudera Manager-managed environment, see
+      <xref audience="integrated" href="cm_ig_install_impala.xml"/><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_impala.html" scope="external" format="html">Installing Impala</xref>.
+    </p>
+
+    <p>
+      Managing your Impala installation through Cloudera Manager has a number of advantages. For example, when you
+      make configuration changes to CDH components using Cloudera Manager, it automatically applies changes to the
+      copies of configuration files, such as <codeph>hive-site.xml</codeph>, that Impala keeps under
+      <filepath>/etc/impala/conf</filepath>. It also sets up the Hive Metastore service that is required for
+      Impala running under CDH 4.1.
+    </p>
+
+    <p>
+      In some cases, depending on the level of Impala, CDH, and Cloudera Manager, you might need to add particular
+      component configuration details in some of the free-form option fields on the Impala configuration pages
+      within Cloudera Manager. <ph conref="../shared/impala_common.xml#common/safety_valve"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_comments.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_comments.xml b/docs/topics/impala_comments.xml
new file mode 100644
index 0000000..07531dc
--- /dev/null
+++ b/docs/topics/impala_comments.xml
@@ -0,0 +1,53 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="comments">
+
+  <title>Comments</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">comments (SQL)</indexterm>
+      Impala supports the familiar styles of SQL comments:
+    </p>
+
+    <ul>
+      <li>
+        All text from a <codeph>--</codeph> sequence to the end of the line is considered a comment and ignored.
+        This type of comment can occur on a single line by itself, or after all or part of a statement.
+      </li>
+
+      <li>
+        All text from a <codeph>/*</codeph> sequence to the next <codeph>*/</codeph> sequence is considered a
+        comment and ignored. This type of comment can stretch over multiple lines. This type of comment can occur
+        on one or more lines by itself, in the middle of a statement, or before or after a statement.
+      </li>
+    </ul>
+
+    <p>
+      For example:
+    </p>
+
+<codeblock>-- This line is a comment about a table.
+create table ...;
+
+/*
+This is a multi-line comment about a query.
+*/
+select ...;
+
+select * from t /* This is an embedded comment about a query. */ where ...;
+
+select * from t -- This is a trailing comment within a multi-line command.
+where ...;
+</codeblock>
+  </conbody>
+</concept>


[39/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_components.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_components.xml b/docs/topics/impala_components.xml
new file mode 100644
index 0000000..44e5c34
--- /dev/null
+++ b/docs/topics/impala_components.xml
@@ -0,0 +1,180 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="intro_components">
+
+  <title>Components of the Impala Server</title>
+  <titlealts audience="PDF"><navtitle>Components</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Concepts"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      The Impala server is a distributed, massively parallel processing (MPP) database engine. It consists of
+      different daemon processes that run on specific hosts within your CDH cluster.
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="intro_impalad">
+
+    <title>The Impala Daemon</title>
+
+    <conbody>
+
+      <p>
+        The core Impala component is a daemon process that runs on each DataNode of the cluster, physically represented
+        by the <codeph>impalad</codeph> process. It reads and writes to data files; accepts queries transmitted
+        from the <codeph>impala-shell</codeph> command, Hue, JDBC, or ODBC; parallelizes the queries and
+        distributes work across the cluster; and transmits intermediate query results back to the
+        central coordinator node.
+      </p>
+
+      <p>
+        You can submit a query to the Impala daemon running on any DataNode, and that instance of the daemon serves as the
+        <term>coordinator node</term> for that query. The other nodes transmit partial results back to the
+        coordinator, which constructs the final result set for a query. When running experiments with functionality
+        through the <codeph>impala-shell</codeph> command, you might always connect to the same Impala daemon for
+        convenience. For clusters running production workloads, you might load-balance by
+        submitting each query to a different Impala daemon in round-robin style, using the JDBC or ODBC interfaces.
+      </p>
+
+      <p>
+        The Impala daemons are in constant communication with the <term>statestore</term>, to confirm which nodes
+        are healthy and can accept new work.
+      </p>
+
+      <p rev="1.2">
+        They also receive broadcast messages from the <cmdname>catalogd</cmdname> daemon (introduced in Impala 1.2)
+        whenever any Impala node in the cluster creates, alters, or drops any type of object, or when an
+        <codeph>INSERT</codeph> or <codeph>LOAD DATA</codeph> statement is processed through Impala. This
+        background communication minimizes the need for <codeph>REFRESH</codeph> or <codeph>INVALIDATE
+        METADATA</codeph> statements that were needed to coordinate metadata across nodes prior to Impala 1.2.
+      </p>
+
+      <p>
+        <b>Related information:</b> <xref href="impala_config_options.xml#config_options"/>,
+        <xref href="impala_processes.xml#processes"/>, <xref href="impala_timeouts.xml#impalad_timeout"/>,
+        <xref href="impala_ports.xml#ports"/>, <xref href="impala_proxy.xml#proxy"/>
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="intro_statestore">
+
+    <title>The Impala Statestore</title>
+
+    <conbody>
+
+      <p>
+        The Impala component known as the <term>statestore</term> checks on the health of Impala daemons on all the
+        DataNodes in a cluster, and continuously relays its findings to each of those daemons. It is physically
+        represented by a daemon process named <codeph>statestored</codeph>; you only need such a process on one
+        host in the cluster. If an Impala daemon goes offline due to hardware failure, network error, software issue,
+        or other reason, the statestore informs all the other Impala daemons so that future queries can avoid making
+        requests to the unreachable node.
+      </p>
+
+      <p>
+        Because the statestore's purpose is to help when things go wrong, it is not critical to the normal
+        operation of an Impala cluster. If the statestore is not running or becomes unreachable, the Impala daemons
+        continue running and distributing work among themselves as usual; the cluster just becomes less robust if
+        other Impala daemons fail while the statestore is offline. When the statestore comes back online, it re-establishes
+        communication with the Impala daemons and resumes its monitoring function.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/statestored_catalogd_ha_blurb"/>
+
+      <p>
+        <b>Related information:</b>
+      </p>
+
+      <p>
+        <xref href="impala_scalability.xml#statestore_scalability"/>,
+        <xref href="impala_config_options.xml#config_options"/>, <xref href="impala_processes.xml#processes"/>,
+        <xref href="impala_timeouts.xml#statestore_timeout"/>, <xref href="impala_ports.xml#ports"/>
+      </p>
+    </conbody>
+  </concept>
+
+  <concept rev="1.2" id="intro_catalogd">
+
+    <title>The Impala Catalog Service</title>
+
+    <conbody>
+
+      <p>
+        The Impala component known as the <term>catalog service</term> relays the metadata changes from Impala SQL
+        statements to all the DataNodes in a cluster. It is physically represented by a daemon process named
+        <codeph>catalogd</codeph>; you only need such a process on one host in the cluster. Because the requests
+        are passed through the statestore daemon, it makes sense to run the <cmdname>statestored</cmdname> and
+        <cmdname>catalogd</cmdname> services on the same host.
+      </p>
+
+      <p>
+        The catalog service avoids the need to issue
+        <codeph>REFRESH</codeph> and <codeph>INVALIDATE METADATA</codeph> statements when the metadata changes are
+        performed by statements issued through Impala. When you create a table, load data, and so on through Hive,
+        you do need to issue <codeph>REFRESH</codeph> or <codeph>INVALIDATE METADATA</codeph> on an Impala node
+        before executing a query there.
+      </p>
+
+      <p>
+        This feature touches a number of aspects of Impala:
+      </p>
+
+<!-- This was formerly a conref, but since the list of links also included a link
+     to this same topic, materializing the list here and removing that
+     circular link. (The conref is still used in Incompatible Changes.)
+
+      <ul conref="../shared/impala_common.xml#common/catalogd_xrefs">
+        <li/>
+      </ul>
+-->
+
+      <ul id="catalogd_xrefs">
+        <li>
+          <p>
+            See <xref href="impala_install.xml#install"/>, <xref href="impala_upgrading.xml#upgrading"/> and
+            <xref href="impala_processes.xml#processes"/>, for usage information for the
+            <cmdname>catalogd</cmdname> daemon.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            The <codeph>REFRESH</codeph> and <codeph>INVALIDATE METADATA</codeph> statements are not needed
+            when the <codeph>CREATE TABLE</codeph>, <codeph>INSERT</codeph>, or other table-changing or
+            data-changing operation is performed through Impala. These statements are still needed if such
+            operations are done through Hive or by manipulating data files directly in HDFS, but in those cases the
+            statements only need to be issued on one Impala node rather than on all nodes. See
+            <xref href="impala_refresh.xml#refresh"/> and
+            <xref href="impala_invalidate_metadata.xml#invalidate_metadata"/> for the latest usage information for
+            those statements.
+          </p>
+        </li>
+      </ul>
+
+      <p conref="../shared/impala_common.xml#common/load_catalog_in_background"/>
+
+      <p conref="../shared/impala_common.xml#common/statestored_catalogd_ha_blurb"/>
+
+      <note>
+        <p conref="../shared/impala_common.xml#common/catalog_server_124"/>
+      </note>
+
+      <p>
+        <b>Related information:</b> <xref href="impala_config_options.xml#config_options"/>,
+        <xref href="impala_processes.xml#processes"/>, <xref href="impala_ports.xml#ports"/>
+      </p>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_compression_codec.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_compression_codec.xml b/docs/topics/impala_compression_codec.xml
new file mode 100644
index 0000000..739c651
--- /dev/null
+++ b/docs/topics/impala_compression_codec.xml
@@ -0,0 +1,98 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="2.0.0" id="compression_codec">
+
+  <title>COMPRESSION_CODEC Query Option (<keyword keyref="impala20"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>COMPRESSION_CODEC</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Compression"/>
+      <data name="Category" value="File Formats"/>
+      <data name="Category" value="Parquet"/>
+      <data name="Category" value="Snappy"/>
+      <data name="Category" value="Gzip"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+<!-- The initial part of this paragraph is copied straight from the #parquet_compression topic. -->
+
+<!-- Could turn into a conref. -->
+
+    <p rev="2.0.0">
+      <indexterm audience="Cloudera">COMPRESSION_CODEC query option</indexterm>
+      When Impala writes Parquet data files using the <codeph>INSERT</codeph> statement, the underlying compression
+      is controlled by the <codeph>COMPRESSION_CODEC</codeph> query option.
+    </p>
+
+    <note>
+      Prior to Impala 2.0, this option was named <codeph>PARQUET_COMPRESSION_CODEC</codeph>. In Impala 2.0 and
+      later, the <codeph>PARQUET_COMPRESSION_CODEC</codeph> name is not recognized. Use the more general name
+      <codeph>COMPRESSION_CODEC</codeph> for new code.
+    </note>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>SET COMPRESSION_CODEC=<varname>codec_name</varname>;</codeblock>
+
+    <p>
+      The allowed values for this query option are <codeph>SNAPPY</codeph> (the default), <codeph>GZIP</codeph>,
+      and <codeph>NONE</codeph>.
+    </p>
+
+    <note>
+      A Parquet file created with <codeph>COMPRESSION_CODEC=NONE</codeph> is still typically smaller than the
+      original data, due to encoding schemes such as run-length encoding and dictionary encoding that are applied
+      separately from compression.
+    </note>
+
+    <p></p>
+
+    <p>
+      The option value is not case-sensitive.
+    </p>
+
+    <p>
+      If the option is set to an unrecognized value, all kinds of queries will fail due to the invalid option
+      setting, not just queries involving Parquet tables. (The value <codeph>BZIP2</codeph> is also recognized, but
+      is not compatible with Parquet tables.)
+    </p>
+
+    <p>
+      <b>Type:</b> <codeph>STRING</codeph>
+    </p>
+
+    <p>
+      <b>Default:</b> <codeph>SNAPPY</codeph>
+    </p>
+
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>set compression_codec=gzip;
+insert into parquet_table_highly_compressed select * from t1;
+
+set compression_codec=snappy;
+insert into parquet_table_compression_plus_fast_queries select * from t1;
+
+set compression_codec=none;
+insert into parquet_table_no_compression select * from t1;
+
+set compression_codec=foo;
+select * from t1 limit 5;
+ERROR: Invalid compression codec: foo
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      For information about how compressing Parquet data files affects query performance, see
+      <xref href="impala_parquet.xml#parquet_compression"/>.
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_compute_stats.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_compute_stats.xml b/docs/topics/impala_compute_stats.xml
new file mode 100644
index 0000000..b915b77
--- /dev/null
+++ b/docs/topics/impala_compute_stats.xml
@@ -0,0 +1,432 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.2.2" id="compute_stats">
+
+  <title>COMPUTE STATS Statement</title>
+  <titlealts audience="PDF"><navtitle>COMPUTE STATS</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Scalability"/>
+      <data name="Category" value="ETL"/>
+      <data name="Category" value="Ingest"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">COMPUTE STATS statement</indexterm>
+      Gathers information about volume and distribution of data in a table and all associated columns and
+      partitions. The information is stored in the metastore database, and used by Impala to help optimize queries.
+      For example, if Impala can determine that a table is large or small, or has many or few distinct values it
+      can organize parallelize the work appropriately for a join query or insert operation. For details about the
+      kinds of information gathered by this statement, see <xref href="impala_perf_stats.xml#perf_stats"/>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock rev="2.1.0">COMPUTE STATS [<varname>db_name</varname>.]<varname>table_name</varname>
+COMPUTE INCREMENTAL STATS [<varname>db_name</varname>.]<varname>table_name</varname> [PARTITION (<varname>partition_spec</varname>)]
+
+<varname>partition_spec</varname> ::= <varname>partition_col</varname>=<varname>constant_value</varname>
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/incremental_partition_spec"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      Originally, Impala relied on users to run the Hive <codeph>ANALYZE TABLE</codeph> statement, but that method
+      of gathering statistics proved unreliable and difficult to use. The Impala <codeph>COMPUTE STATS</codeph>
+      statement is built from the ground up to improve the reliability and user-friendliness of this operation.
+      <codeph>COMPUTE STATS</codeph> does not require any setup steps or special configuration. You only run a
+      single Impala <codeph>COMPUTE STATS</codeph> statement to gather both table and column statistics, rather
+      than separate Hive <codeph>ANALYZE TABLE</codeph> statements for each kind of statistics.
+    </p>
+
+    <p rev="2.1.0">
+      The <codeph>COMPUTE INCREMENTAL STATS</codeph> variation is a shortcut for partitioned tables that works on a
+      subset of partitions rather than the entire table. The incremental nature makes it suitable for large tables
+      with many partitions, where a full <codeph>COMPUTE STATS</codeph> operation takes too long to be practical
+      each time a partition is added or dropped. See <xref href="impala_perf_stats.xml#perf_stats_incremental"/>
+      for full usage details.
+    </p>
+
+    <p>
+      <codeph>COMPUTE INCREMENTAL STATS</codeph> only applies to partitioned tables. If you use the
+      <codeph>INCREMENTAL</codeph> clause for an unpartitioned table, Impala automatically uses the original
+      <codeph>COMPUTE STATS</codeph> statement. Such tables display <codeph>false</codeph> under the
+      <codeph>Incremental stats</codeph> column of the <codeph>SHOW TABLE STATS</codeph> output.
+    </p>
+
+    <note>
+      Because many of the most performance-critical and resource-intensive operations rely on table and column
+      statistics to construct accurate and efficient plans, <codeph>COMPUTE STATS</codeph> is an important step at
+      the end of your ETL process. Run <codeph>COMPUTE STATS</codeph> on all tables as your first step during
+      performance tuning for slow queries, or troubleshooting for out-of-memory conditions:
+      <ul>
+        <li>
+          Accurate statistics help Impala construct an efficient query plan for join queries, improving performance
+          and reducing memory usage.
+        </li>
+
+        <li>
+          Accurate statistics help Impala distribute the work effectively for insert operations into Parquet
+          tables, improving performance and reducing memory usage.
+        </li>
+
+        <li rev="1.3.0">
+          Accurate statistics help Impala estimate the memory required for each query, which is important when you
+          use resource management features, such as admission control and the YARN resource management framework.
+          The statistics help Impala to achieve high concurrency, full utilization of available memory, and avoid
+          contention with workloads from other Hadoop components.
+        </li>
+      </ul>
+    </note>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+    <p rev="2.3.0">
+      Currently, the statistics created by the <codeph>COMPUTE STATS</codeph> statement do not include
+      information about complex type columns. The column stats metrics for complex columns are always shown
+      as -1. For queries involving complex type columns, Impala uses
+      heuristics to estimate the data distribution within such columns.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/hbase_blurb"/>
+
+    <p>
+      <codeph>COMPUTE STATS</codeph> works for HBase tables also. The statistics gathered for HBase tables are
+      somewhat different than for HDFS-backed tables, but that metadata is still used for optimization when HBase
+      tables are involved in join queries.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/s3_blurb"/>
+
+    <p rev="2.2.0">
+      <codeph>COMPUTE STATS</codeph> also works for tables where data resides in the Amazon Simple Storage Service (S3).
+      See <xref href="impala_s3.xml#s3"/> for details.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/performance_blurb"/>
+
+    <p>
+      The statistics collected by <codeph>COMPUTE STATS</codeph> are used to optimize join queries
+      <codeph>INSERT</codeph> operations into Parquet tables, and other resource-intensive kinds of SQL statements.
+      See <xref href="impala_perf_stats.xml#perf_stats"/> for details.
+    </p>
+
+    <p>
+      For large tables, the <codeph>COMPUTE STATS</codeph> statement itself might take a long time and you
+      might need to tune its performance. The <codeph>COMPUTE STATS</codeph> statement does not work with the
+      <codeph>EXPLAIN</codeph> statement, or the <codeph>SUMMARY</codeph> command in <cmdname>impala-shell</cmdname>.
+      You can use the <codeph>PROFILE</codeph> statement in <cmdname>impala-shell</cmdname> to examine timing information
+      for the statement as a whole. If a basic <codeph>COMPUTE STATS</codeph> statement takes a long time for a
+      partitioned table, consider switching to the <codeph>COMPUTE INCREMENTAL STATS</codeph> syntax so that only
+      newly added partitions are analyzed each time.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      This example shows two tables, <codeph>T1</codeph> and <codeph>T2</codeph>, with a small number distinct
+      values linked by a parent-child relationship between <codeph>T1.ID</codeph> and <codeph>T2.PARENT</codeph>.
+      <codeph>T1</codeph> is tiny, while <codeph>T2</codeph> has approximately 100K rows. Initially, the statistics
+      includes physical measurements such as the number of files, the total size, and size measurements for
+      fixed-length columns such as with the <codeph>INT</codeph> type. Unknown values are represented by -1. After
+      running <codeph>COMPUTE STATS</codeph> for each table, much more information is available through the
+      <codeph>SHOW STATS</codeph> statements. If you were running a join query involving both of these tables, you
+      would need statistics for both tables to get the most effective optimization for the query.
+    </p>
+
+<!-- Note: chopped off any excess characters at position 87 and after,
+           to avoid weird wrapping in PDF.
+           Applies to any subsequent examples with output from SHOW ... STATS too. -->
+
+<codeblock>[localhost:21000] &gt; show table stats t1;
+Query: show table stats t1
++-------+--------+------+--------+
+| #Rows | #Files | Size | Format |
++-------+--------+------+--------+
+| -1    | 1      | 33B  | TEXT   |
++-------+--------+------+--------+
+Returned 1 row(s) in 0.02s
+[localhost:21000] &gt; show table stats t2;
+Query: show table stats t2
++-------+--------+----------+--------+
+| #Rows | #Files | Size     | Format |
++-------+--------+----------+--------+
+| -1    | 28     | 960.00KB | TEXT   |
++-------+--------+----------+--------+
+Returned 1 row(s) in 0.01s
+[localhost:21000] &gt; show column stats t1;
+Query: show column stats t1
++--------+--------+------------------+--------+----------+----------+
+| Column | Type   | #Distinct Values | #Nulls | Max Size | Avg Size |
++--------+--------+------------------+--------+----------+----------+
+| id     | INT    | -1               | -1     | 4        | 4        |
+| s      | STRING | -1               | -1     | -1       | -1       |
++--------+--------+------------------+--------+----------+----------+
+Returned 2 row(s) in 1.71s
+[localhost:21000] &gt; show column stats t2;
+Query: show column stats t2
++--------+--------+------------------+--------+----------+----------+
+| Column | Type   | #Distinct Values | #Nulls | Max Size | Avg Size |
++--------+--------+------------------+--------+----------+----------+
+| parent | INT    | -1               | -1     | 4        | 4        |
+| s      | STRING | -1               | -1     | -1       | -1       |
++--------+--------+------------------+--------+----------+----------+
+Returned 2 row(s) in 0.01s
+[localhost:21000] &gt; compute stats t1;
+Query: compute stats t1
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 1 partition(s) and 2 column(s). |
++-----------------------------------------+
+Returned 1 row(s) in 5.30s
+[localhost:21000] &gt; show table stats t1;
+Query: show table stats t1
++-------+--------+------+--------+
+| #Rows | #Files | Size | Format |
++-------+--------+------+--------+
+| 3     | 1      | 33B  | TEXT   |
++-------+--------+------+--------+
+Returned 1 row(s) in 0.01s
+[localhost:21000] &gt; show column stats t1;
+Query: show column stats t1
++--------+--------+------------------+--------+----------+----------+
+| Column | Type   | #Distinct Values | #Nulls | Max Size | Avg Size |
++--------+--------+------------------+--------+----------+----------+
+| id     | INT    | 3                | -1     | 4        | 4        |
+| s      | STRING | 3                | -1     | -1       | -1       |
++--------+--------+------------------+--------+----------+----------+
+Returned 2 row(s) in 0.02s
+[localhost:21000] &gt; compute stats t2;
+Query: compute stats t2
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 1 partition(s) and 2 column(s). |
++-----------------------------------------+
+Returned 1 row(s) in 5.70s
+[localhost:21000] &gt; show table stats t2;
+Query: show table stats t2
++-------+--------+----------+--------+
+| #Rows | #Files | Size     | Format |
++-------+--------+----------+--------+
+| 98304 | 1      | 960.00KB | TEXT   |
++-------+--------+----------+--------+
+Returned 1 row(s) in 0.03s
+[localhost:21000] &gt; show column stats t2;
+Query: show column stats t2
++--------+--------+------------------+--------+----------+----------+
+| Column | Type   | #Distinct Values | #Nulls | Max Size | Avg Size |
++--------+--------+------------------+--------+----------+----------+
+| parent | INT    | 3                | -1     | 4        | 4        |
+| s      | STRING | 6                | -1     | 14       | 9.3      |
++--------+--------+------------------+--------+----------+----------+
+Returned 2 row(s) in 0.01s</codeblock>
+
+    <p rev="2.1.0">
+      The following example shows how to use the <codeph>INCREMENTAL</codeph> clause, available in Impala 2.1.0 and
+      higher. The <codeph>COMPUTE INCREMENTAL STATS</codeph> syntax lets you collect statistics for newly added or
+      changed partitions, without rescanning the entire table.
+    </p>
+
+<codeblock>-- Initially the table has no incremental stats, as indicated
+-- by -1 under #Rows and false under Incremental stats.
+show table stats item_partitioned;
++-------------+-------+--------+----------+--------------+---------+------------------
+| i_category  | #Rows | #Files | Size     | Bytes Cached | Format  | Incremental stats
++-------------+-------+--------+----------+--------------+---------+------------------
+| Books       | -1    | 1      | 223.74KB | NOT CACHED   | PARQUET | false
+| Children    | -1    | 1      | 230.05KB | NOT CACHED   | PARQUET | false
+| Electronics | -1    | 1      | 232.67KB | NOT CACHED   | PARQUET | false
+| Home        | -1    | 1      | 232.56KB | NOT CACHED   | PARQUET | false
+| Jewelry     | -1    | 1      | 223.72KB | NOT CACHED   | PARQUET | false
+| Men         | -1    | 1      | 231.25KB | NOT CACHED   | PARQUET | false
+| Music       | -1    | 1      | 237.90KB | NOT CACHED   | PARQUET | false
+| Shoes       | -1    | 1      | 234.90KB | NOT CACHED   | PARQUET | false
+| Sports      | -1    | 1      | 227.97KB | NOT CACHED   | PARQUET | false
+| Women       | -1    | 1      | 226.27KB | NOT CACHED   | PARQUET | false
+| Total       | -1    | 10     | 2.25MB   | 0B           |         |
++-------------+-------+--------+----------+--------------+---------+------------------
+
+-- After the first COMPUTE INCREMENTAL STATS,
+-- all partitions have stats.
+compute incremental stats item_partitioned;
++-------------------------------------------+
+| summary                                   |
++-------------------------------------------+
+| Updated 10 partition(s) and 21 column(s). |
++-------------------------------------------+
+show table stats item_partitioned;
++-------------+-------+--------+----------+--------------+---------+------------------
+| i_category  | #Rows | #Files | Size     | Bytes Cached | Format  | Incremental stats
++-------------+-------+--------+----------+--------------+---------+------------------
+| Books       | 1733  | 1      | 223.74KB | NOT CACHED   | PARQUET | true
+| Children    | 1786  | 1      | 230.05KB | NOT CACHED   | PARQUET | true
+| Electronics | 1812  | 1      | 232.67KB | NOT CACHED   | PARQUET | true
+| Home        | 1807  | 1      | 232.56KB | NOT CACHED   | PARQUET | true
+| Jewelry     | 1740  | 1      | 223.72KB | NOT CACHED   | PARQUET | true
+| Men         | 1811  | 1      | 231.25KB | NOT CACHED   | PARQUET | true
+| Music       | 1860  | 1      | 237.90KB | NOT CACHED   | PARQUET | true
+| Shoes       | 1835  | 1      | 234.90KB | NOT CACHED   | PARQUET | true
+| Sports      | 1783  | 1      | 227.97KB | NOT CACHED   | PARQUET | true
+| Women       | 1790  | 1      | 226.27KB | NOT CACHED   | PARQUET | true
+| Total       | 17957 | 10     | 2.25MB   | 0B           |         |
++-------------+-------+--------+----------+--------------+---------+------------------
+
+-- Add a new partition...
+alter table item_partitioned add partition (i_category='Camping');
+-- Add or replace files in HDFS outside of Impala,
+-- rendering the stats for a partition obsolete.
+!import_data_into_sports_partition.sh
+refresh item_partitioned;
+drop incremental stats item_partitioned partition (i_category='Sports');
+-- Now some partitions have incremental stats
+-- and some do not.
+show table stats item_partitioned;
++-------------+-------+--------+----------+--------------+---------+------------------
+| i_category  | #Rows | #Files | Size     | Bytes Cached | Format  | Incremental stats
++-------------+-------+--------+----------+--------------+---------+------------------
+| Books       | 1733  | 1      | 223.74KB | NOT CACHED   | PARQUET | true
+| Camping     | -1    | 1      | 408.02KB | NOT CACHED   | PARQUET | false
+| Children    | 1786  | 1      | 230.05KB | NOT CACHED   | PARQUET | true
+| Electronics | 1812  | 1      | 232.67KB | NOT CACHED   | PARQUET | true
+| Home        | 1807  | 1      | 232.56KB | NOT CACHED   | PARQUET | true
+| Jewelry     | 1740  | 1      | 223.72KB | NOT CACHED   | PARQUET | true
+| Men         | 1811  | 1      | 231.25KB | NOT CACHED   | PARQUET | true
+| Music       | 1860  | 1      | 237.90KB | NOT CACHED   | PARQUET | true
+| Shoes       | 1835  | 1      | 234.90KB | NOT CACHED   | PARQUET | true
+| Sports      | -1    | 1      | 227.97KB | NOT CACHED   | PARQUET | false
+| Women       | 1790  | 1      | 226.27KB | NOT CACHED   | PARQUET | true
+| Total       | 17957 | 11     | 2.65MB   | 0B           |         |
++-------------+-------+--------+----------+--------------+---------+------------------
+
+-- After another COMPUTE INCREMENTAL STATS,
+-- all partitions have incremental stats, and only the 2
+-- partitions without incremental stats were scanned.
+compute incremental stats item_partitioned;
++------------------------------------------+
+| summary                                  |
++------------------------------------------+
+| Updated 2 partition(s) and 21 column(s). |
++------------------------------------------+
+show table stats item_partitioned;
++-------------+-------+--------+----------+--------------+---------+------------------
+| i_category  | #Rows | #Files | Size     | Bytes Cached | Format  | Incremental stats
++-------------+-------+--------+----------+--------------+---------+------------------
+| Books       | 1733  | 1      | 223.74KB | NOT CACHED   | PARQUET | true
+| Camping     | 5328  | 1      | 408.02KB | NOT CACHED   | PARQUET | true
+| Children    | 1786  | 1      | 230.05KB | NOT CACHED   | PARQUET | true
+| Electronics | 1812  | 1      | 232.67KB | NOT CACHED   | PARQUET | true
+| Home        | 1807  | 1      | 232.56KB | NOT CACHED   | PARQUET | true
+| Jewelry     | 1740  | 1      | 223.72KB | NOT CACHED   | PARQUET | true
+| Men         | 1811  | 1      | 231.25KB | NOT CACHED   | PARQUET | true
+| Music       | 1860  | 1      | 237.90KB | NOT CACHED   | PARQUET | true
+| Shoes       | 1835  | 1      | 234.90KB | NOT CACHED   | PARQUET | true
+| Sports      | 1783  | 1      | 227.97KB | NOT CACHED   | PARQUET | true
+| Women       | 1790  | 1      | 226.27KB | NOT CACHED   | PARQUET | true
+| Total       | 17957 | 11     | 2.65MB   | 0B           |         |
++-------------+-------+--------+----------+--------------+---------+------------------
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/file_format_blurb"/>
+
+    <p>
+      The <codeph>COMPUTE STATS</codeph> statement works with tables created with any of the file formats supported
+      by Impala. See <xref href="impala_file_formats.xml#file_formats"/> for details about working with the
+      different file formats. The following considerations apply to <codeph>COMPUTE STATS</codeph> depending on the
+      file format of the table.
+    </p>
+
+    <p>
+      The <codeph>COMPUTE STATS</codeph> statement works with text tables with no restrictions. These tables can be
+      created through either Impala or Hive.
+    </p>
+
+    <p>
+      The <codeph>COMPUTE STATS</codeph> statement works with Parquet tables. These tables can be created through
+      either Impala or Hive.
+    </p>
+
+    <p>
+      The <codeph>COMPUTE STATS</codeph> statement works with Avro tables without restriction in CDH 5.4 / Impala 2.2
+      and higher. In earlier releases, <codeph>COMPUTE STATS</codeph> worked only for Avro tables created through Hive,
+      and required the <codeph>CREATE TABLE</codeph> statement to use SQL-style column names and types rather than an
+      Avro-style schema specification.
+    </p>
+
+    <p>
+      The <codeph>COMPUTE STATS</codeph> statement works with RCFile tables with no restrictions. These tables can
+      be created through either Impala or Hive.
+    </p>
+
+    <p>
+      The <codeph>COMPUTE STATS</codeph> statement works with SequenceFile tables with no restrictions. These
+      tables can be created through either Impala or Hive.
+    </p>
+
+    <p>
+      The <codeph>COMPUTE STATS</codeph> statement works with partitioned tables, whether all the partitions use
+      the same file format, or some partitions are defined through <codeph>ALTER TABLE</codeph> to use different
+      file formats.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_maybe"/>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/decimal_no_stats"/>
+
+    <note conref="../shared/impala_common.xml#common/compute_stats_nulls"/>
+
+    <p conref="../shared/impala_common.xml#common/internals_blurb"/>
+    <p>
+      Behind the scenes, the <codeph>COMPUTE STATS</codeph> statement
+      executes two statements: one to count the rows of each partition
+      in the table (or the entire table if unpartitioned) through the
+      <codeph>COUNT(*)</codeph> function,
+      and another to count the approximate number of distinct values
+      in each column through the <codeph>NDV()</codeph> function.
+      You might see these queries in your monitoring and diagnostic displays.
+      The same factors that affect the performance, scalability, and
+      execution of other queries (such as parallel execution, memory usage,
+      admission control, and timeouts) also apply to the queries run by the
+      <codeph>COMPUTE STATS</codeph> statement.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+    <p rev="CDH-19187">
+      The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+      typically the <codeph>impala</codeph> user, must have read
+      permission for all affected files in the source directory:
+      all files in the case of an unpartitioned table or
+      a partitioned table in the case of <codeph>COMPUTE STATS</codeph>;
+      or all the files in partitions without incremental stats in
+      the case of <codeph>COMPUTE INCREMENTAL STATS</codeph>.
+      It must also have read and execute permissions for all
+      relevant directories holding the data files.
+      (Essentially, <codeph>COMPUTE STATS</codeph> requires the
+      same permissions as the underlying <codeph>SELECT</codeph> queries it runs
+      against the table.)
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_drop_stats.xml#drop_stats"/>, <xref href="impala_show.xml#show_table_stats"/>,
+      <xref href="impala_show.xml#show_column_stats"/>, <xref href="impala_perf_stats.xml#perf_stats"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_concepts.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_concepts.xml b/docs/topics/impala_concepts.xml
new file mode 100644
index 0000000..74c1016
--- /dev/null
+++ b/docs/topics/impala_concepts.xml
@@ -0,0 +1,296 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="concepts">
+
+  <title>Impala Concepts and Architecture</title>
+  <titlealts audience="PDF"><navtitle>Concepts and Architecture</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Concepts"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Stub Pages"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+    <draft-comment author="-dita-use-conref-target" audience="integrated"
+      conref="../shared/cdh_cm_common.xml#id_dgz_rhr_kv/draft-comment-test"/>
+
+    <p>
+      The following sections provide background information to help you become productive using Impala and
+      its features. Where appropriate, the explanations include context to help understand how aspects of Impala
+      relate to other technologies you might already be familiar with, such as relational database management
+      systems and data warehouses, or other Hadoop components such as Hive, HDFS, and HBase.
+    </p>
+
+    <p outputclass="toc"/>
+  </conbody>
+
+<!-- These other topics are waiting to be filled in. Could become subtopics or top-level topics depending on the depth of coverage in each case. -->
+
+  <concept id="intro_data_lifecycle" audience="Cloudera">
+
+    <title>Overview of the Data Lifecycle for Impala</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="intro_etl" audience="Cloudera">
+
+    <title>Overview of the Extract, Transform, Load (ETL) Process for Impala</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="ETL"/>
+      <data name="Category" value="Ingest"/>
+      <data name="Category" value="Concepts"/>
+    </metadata>
+  </prolog>
+
+    <conbody/>
+  </concept>
+
+  <concept id="intro_hadoop_data" audience="Cloudera">
+
+    <title>How Impala Works with Hadoop Data Files</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="intro_web_ui" audience="Cloudera">
+
+    <title>Overview of the Impala Web Interface</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="intro_bi" audience="Cloudera">
+
+    <title>Using Impala with Business Intelligence Tools</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="intro_ha" audience="Cloudera">
+
+    <title>Overview of Impala Availability and Fault Tolerance</title>
+
+    <conbody/>
+  </concept>
+
+<!-- This is pretty much ready to go. Decide if it should go under "Concepts" or "Performance",
+     and if it should be split out into a separate file, and then take out the audience= attribute
+     to make it visible.
+-->
+
+  <concept id="intro_llvm" audience="Cloudera">
+
+    <title>Overview of Impala Runtime Code Generation</title>
+
+    <conbody>
+
+<!-- Adapted from the CIDR15 paper written by the Impala team. -->
+
+      <p>
+        Impala uses <term>LLVM</term> (a compiler library and collection of related tools) to perform just-in-time
+        (JIT) compilation within the running <cmdname>impalad</cmdname> process. This runtime code generation
+        technique improves query execution times by generating native code optimized for the architecture of each
+        host in your particular cluster. Performance gains of 5 times or more are typical for representative
+        workloads.
+      </p>
+
+      <p>
+        Impala uses runtime code generation to produce query-specific versions of functions that are critical to
+        performance. In particular, code generation is applied to <term>inner loop</term> functions, that is, those
+        that are executed many times (for every tuple) in a given query, and thus constitute a large portion of the
+        total time the query takes to execute. For example, when Impala scans a data file, it calls a function to
+        parse each record into Impala\u2019s in-memory tuple format. For queries scanning large tables, billions of
+        records could result in billions of function calls. This function must therefore be extremely efficient for
+        good query performance, and removing even a few instructions from each function call can result in large
+        query speedups.
+      </p>
+
+      <p>
+        Overall, JIT compilation has an effect similar to writing custom code to process a query. For example, it
+        eliminates branches, unrolls loops, propagates constants, offsets and pointers, and inlines functions.
+        Inlining is especially valuable for functions used internally to evaluate expressions, where the function
+        call itself is more expensive than the function body (for example, a function that adds two numbers).
+        Inlining functions also increases instruction-level parallelism, and allows the compiler to make further
+        optimizations such as subexpression elimination across expressions.
+      </p>
+
+      <p>
+        Impala generates runtime query code automatically, so you do not need to do anything special to get this
+        performance benefit. This technique is most effective for complex and long-running queries that process
+        large numbers of rows. If you need to issue a series of short, small queries, you might turn off this
+        feature to avoid the overhead of compilation time for each query. In this case, issue the statement
+        <codeph>SET DISABLE_CODEGEN=true</codeph> to turn off runtime code generation for the duration of the
+        current session.
+      </p>
+
+<!--
+      <p>
+        Without code generation,
+        functions tend to be suboptimal
+        to handle situations that cannot be predicted in advance.
+        For example,
+        a record-parsing function that
+        only handles integer types will be faster at parsing an integer-only file
+        than a function that handles other data types
+        such as strings and floating-point numbers.
+        However, the schemas of the files to
+        be scanned are unknown at compile time,
+        and so a general-purpose function must be used, even if at runtime
+        it is known that more limited functionality is sufficient.
+      </p>
+
+      <p>
+        A source of large runtime overheads are virtual functions. Virtual function calls incur a large performance
+        penalty, particularly when the called function is very simple, as the calls cannot be inlined.
+        If the type of the object instance is known at runtime, we can use code generation to replace the virtual
+        function call with a call directly to the correct function, which can then be inlined. This is especially
+        valuable when evaluating expression trees. In Impala (as in many systems), expressions are composed of a
+        tree of individual operators and functions.
+      </p>
+
+      <p>
+        Each type of expression that can appear in a query is implemented internally by overriding a virtual function.
+        Many of these expression functions are quite simple, for example, adding two numbers.
+        The virtual function call can be more expensive than the function body itself. By resolving the virtual
+        function calls with code generation and then inlining the resulting function calls, Impala can evaluate expressions
+        directly with no function call overhead. Inlining functions also increases
+        instruction-level parallelism, and allows the compiler to make further optimizations such as subexpression
+        elimination across expressions.
+      </p>
+-->
+    </conbody>
+  </concept>
+
+<!-- Same as the previous section: adapted from CIDR paper, ready to externalize after deciding where to go. -->
+
+  <concept audience="Cloudera" id="intro_io">
+
+    <title>Overview of Impala I/O</title>
+
+    <conbody>
+
+      <p>
+        Efficiently retrieving data from HDFS is a challenge for all SQL-on-Hadoop systems. To perform
+        data scans from both disk and memory at or near hardware speed, Impala uses an HDFS feature called
+        <term>short-circuit local reads</term> to bypass the DataNode protocol when reading from local disk. Impala
+        can read at almost disk bandwidth (approximately 100 MB/s per disk) and is typically able to saturate all
+        available disks. For example, with 12 disks, Impala is typically capable of sustaining I/O at 1.2 GB/sec.
+        Furthermore, <term>HDFS caching</term> allows Impala to access memory-resident data at memory bus speed,
+        and saves CPU cycles as there is no need to copy or checksum data blocks within memory.
+      </p>
+
+      <p>
+        The I/O manager component interfaces with storage devices to read and write data. I/O manager assigns a
+        fixed number of worker threads per physical disk (currently one thread per rotational disk and eight per
+        SSD), providing an asynchronous interface to clients (<term>scanner threads</term>).
+      </p>
+    </conbody>
+  </concept>
+
+<!-- Same as the previous section: adapted from CIDR paper, ready to externalize after deciding where to go. -->
+
+<!-- Although good idea to get some answers from Henry first. -->
+
+  <concept audience="Cloudera" id="intro_state_distribution">
+
+    <title>State distribution</title>
+
+    <conbody>
+
+      <p>
+        As a massively parallel database that can run on hundreds of nodes, Impala must coordinate and synchronize
+        its metadata across the entire cluster. Impala's symmetric-node architecture means that any node can accept
+        and execute queries, and thus each node needs up-to-date versions of the system catalog and a knowledge of
+        which hosts the <cmdname>impalad</cmdname> daemons run on. To avoid the overhead of TCP connections and
+        remote procedure calls to retrieve metadata during query planning, Impala implements a simple
+        publish-subscribe service called the <term>statestore</term> to push metadata changes to a set of
+        subscribers (the <cmdname>impalad</cmdname> daemons running on all the DataNodes).
+      </p>
+
+      <p>
+        The statestore maintains a set of topics, which are arrays of <codeph>(<varname>key</varname>,
+        <varname>value</varname>, <varname>version</varname>)</codeph> triplets called <term>entries</term> where
+        <varname>key</varname> and <varname>value</varname> are byte arrays, and <varname>version</varname> is a
+        64-bit integer. A topic is defined by an application, and so the statestore has no understanding of the
+        contents of any topic entry. Topics are persistent through the lifetime of the statestore, but are not
+        persisted across service restarts. Processes that receive updates to any topic are called
+        <term>subscribers</term>, and express their interest by registering with the statestore at startup and
+        providing a list of topics. The statestore responds to registration by sending the subscriber an initial
+        topic update for each registered topic, which consists of all the entries currently in that topic.
+      </p>
+
+<!-- Henry: OK, but in practice, what is in these topic messages for Impala? -->
+
+      <p>
+        After registration, the statestore periodically sends two kinds of messages to each subscriber. The first
+        kind of message is a topic update, and consists of all changes to a topic (new entries, modified entries
+        and deletions) since the last update was successfully sent to the subscriber. Each subscriber maintains a
+        per-topic most-recent-version identifier which allows the statestore to only send the delta between
+        updates. In response to a topic update, each subscriber sends a list of changes it intends to make to its
+        subscribed topics. Those changes are guaranteed to have been applied by the time the next update is
+        received.
+      </p>
+
+      <p>
+        The second kind of statestore message is a <term>heartbeat</term>, formerly sometimes called
+        <term>keepalive</term>. The statestore uses heartbeat messages to maintain the connection to each
+        subscriber, which would otherwise time out its subscription and attempt to re-register.
+      </p>
+
+      <p>
+        Prior to Impala 2.0, both kinds of communication were combined in a single kind of message. Because these
+        messages could be very large in instances with thousands of tables, partitions, data files, and so on,
+        Impala 2.0 and higher divides the types of messages so that the small heartbeat pings can be transmitted
+        and acknowledged quickly, increasing the reliability of the statestore mechanism that detects when Impala
+        nodes become unavailable.
+      </p>
+
+      <p>
+        If the statestore detects a failed subscriber (for example, by repeated failed heartbeat deliveries), it
+        stops sending updates to that node.
+<!-- Henry: what are examples of these transient topic entries? -->
+        Some topic entries are marked as transient, meaning that if their owning subscriber fails, they are
+        removed.
+      </p>
+
+      <p>
+        Although the asynchronous nature of this mechanism means that metadata updates might take some time to
+        propagate across the entire cluster, that does not affect the consistency of query planning or results.
+        Each query is planned and coordinated by a particular node, so as long as the coordinator node is aware of
+        the existence of the relevant tables, data files, and so on, it can distribute the query work to other
+        nodes even if those other nodes have not received the latest metadata updates.
+<!-- Henry: need another example here of what's in a topic, e.g. is it the list of available tables? -->
+<!--
+        For example, query planning is performed on a single node based on the
+        catalog metadata topic, and once a full plan has been computed, all information required to execute that
+        plan is distributed directly to the executing nodes.
+        There is no requirement that an executing node should
+        know about the same version of the catalog metadata topic.
+-->
+      </p>
+
+      <p>
+        We have found that the statestore process with default settings scales well to medium sized clusters, and
+        can serve our largest deployments with some configuration changes.
+<!-- Henry: elaborate on the configuration changes. -->
+      </p>
+
+      <p>
+<!-- Henry: other examples like load information? How is load information used? -->
+        The statestore does not persist any metadata to disk: all current metadata is pushed to the statestore by
+        its subscribers (for example, load information). Therefore, should a statestore restart, its state can be
+        recovered during the initial subscriber registration phase. Or if the machine that the statestore is
+        running on fails, a new statestore process can be started elsewhere, and subscribers can fail over to it.
+        There is no built-in failover mechanism in Impala, instead deployments commonly use a retargetable DNS
+        entry to force subscribers to automatically move to the new process instance.
+<!-- Henry: translate that last sentence into instructions / guidelines. -->
+      </p>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_conditional_functions.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_conditional_functions.xml b/docs/topics/impala_conditional_functions.xml
new file mode 100644
index 0000000..23de779
--- /dev/null
+++ b/docs/topics/impala_conditional_functions.xml
@@ -0,0 +1,443 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="conditional_functions">
+
+  <title>Impala Conditional Functions</title>
+  <titlealts audience="PDF"><navtitle>Conditional Functions</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Querying"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Impala supports the following conditional functions for testing equality, comparison operators, and nullity:
+    </p>
+
+    <dl>
+      <dlentry id="case">
+
+        <dt>
+          <codeph>CASE a WHEN b THEN c [WHEN d THEN e]... [ELSE f] END</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">CASE expression</indexterm>
+          <b>Purpose:</b> Compares an expression to one or more possible values, and returns a corresponding result
+          when a match is found.
+          <p conref="../shared/impala_common.xml#common/return_same_type"/>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            In this form of the <codeph>CASE</codeph> expression, the initial value <codeph>A</codeph>
+            being evaluated for each row it typically a column reference, or an expression involving
+            a column. This form can only compare against a set of specified values, not ranges,
+            multi-value comparisons such as <codeph>BETWEEN</codeph> or <codeph>IN</codeph>,
+            regular expressions, or <codeph>NULL</codeph>.
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            Although this example is split across multiple lines, you can put any or all parts of a <codeph>CASE</codeph> expression
+            on a single line, with no punctuation or other separators between the <codeph>WHEN</codeph>,
+            <codeph>ELSE</codeph>, and <codeph>END</codeph> clauses.
+          </p>
+<codeblock>select case x
+    when 1 then 'one'
+    when 2 then 'two'
+    when 0 then 'zero'
+    else 'out of range'
+  end
+    from t1;
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="case2">
+
+        <dt>
+          <codeph>CASE WHEN a THEN b [WHEN c THEN d]... [ELSE e] END</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">CASE expression</indexterm>
+          <b>Purpose:</b> Tests whether any of a sequence of expressions is true, and returns a corresponding
+          result for the first true expression.
+          <p conref="../shared/impala_common.xml#common/return_same_type"/>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            <codeph>CASE</codeph> expressions without an initial test value have more flexibility.
+            For example, they can test different columns in different <codeph>WHEN</codeph> clauses,
+            or use comparison operators such as <codeph>BETWEEN</codeph>, <codeph>IN</codeph> and <codeph>IS NULL</codeph>
+            rather than comparing against discrete values.
+          </p>
+          <p>
+            <codeph>CASE</codeph> expressions are often the foundation of long queries that
+            summarize and format results for easy-to-read reports. For example, you might
+            use a <codeph>CASE</codeph> function call to turn values from a numeric column
+            into category strings corresponding to integer values, or labels such as <q>Small</q>,
+            <q>Medium</q> and <q>Large</q> based on ranges. Then subsequent parts of the
+            query might aggregate based on the transformed values, such as how many
+            values are classified as small, medium, or large. You can also use <codeph>CASE</codeph>
+            to signal problems with out-of-bounds values, <codeph>NULL</codeph> values,
+            and so on.
+          </p>
+          <p>
+            By using operators such as <codeph>OR</codeph>, <codeph>IN</codeph>,
+            <codeph>REGEXP</codeph>, and so on in <codeph>CASE</codeph> expressions,
+            you can build extensive tests and transformations into a single query.
+            Therefore, applications that construct SQL statements often rely heavily on <codeph>CASE</codeph>
+            calls in the generated SQL code.
+          </p>
+          <p>
+            Because this flexible form of the <codeph>CASE</codeph> expressions allows you to perform
+            many comparisons and call multiple functions when evaluating each row, be careful applying
+            elaborate <codeph>CASE</codeph> expressions to queries that process large amounts of data.
+            For example, when practical, evaluate and transform values through <codeph>CASE</codeph>
+            after applying operations such as aggregations that reduce the size of the result set;
+            transform numbers to strings after performing joins with the original numeric values.
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            Although this example is split across multiple lines, you can put any or all parts of a <codeph>CASE</codeph> expression
+            on a single line, with no punctuation or other separators between the <codeph>WHEN</codeph>,
+            <codeph>ELSE</codeph>, and <codeph>END</codeph> clauses.
+          </p>
+<codeblock>select case
+    when dayname(now()) in ('Saturday','Sunday') then 'result undefined on weekends'
+    when x > y then 'x greater than y'
+    when x = y then 'x and y are equal'
+    when x is null or y is null then 'one of the columns is null'
+    else null
+  end
+    from t1;
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="coalesce">
+
+        <dt>
+          <codeph>coalesce(type v1, type v2, ...)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">coalesce() function</indexterm>
+          <b>Purpose:</b> Returns the first specified argument that is not <codeph>NULL</codeph>, or
+          <codeph>NULL</codeph> if all arguments are <codeph>NULL</codeph>.
+          <p conref="../shared/impala_common.xml#common/return_same_type"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="2.0.0" id="decode">
+
+        <dt>
+          <codeph>decode(type expression, type search1, type result1 [, type search2, type result2 ...] [, type
+          default] )</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">decode() function</indexterm>
+          <b>Purpose:</b> Compares an expression to one or more possible values, and returns a corresponding result
+          when a match is found.
+          <p conref="../shared/impala_common.xml#common/return_same_type"/>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            Can be used as shorthand for a <codeph>CASE</codeph> expression.
+          </p>
+          <p>
+            The original expression and the search expressions must of the same type or convertible types. The
+            result expression can be a different type, but all result expressions must be of the same type.
+          </p>
+          <p>
+            Returns a successful match If the original expression is <codeph>NULL</codeph> and a search expression
+            is also <codeph>NULL</codeph>. the
+          </p>
+          <p>
+            Returns <codeph>NULL</codeph> if the final <codeph>default</codeph> value is omitted and none of the
+            search expressions match the original expression.
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following example translates numeric day values into descriptive names:
+          </p>
+<codeblock>SELECT event, decode(day_of_week, 1, "Monday", 2, "Tuesday", 3, "Wednesday",
+  4, "Thursday", 5, "Friday", 6, "Saturday", 7, "Sunday", "Unknown day")
+  FROM calendar;
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="if">
+
+        <dt>
+          <codeph>if(boolean condition, type ifTrue, type ifFalseOrNull)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">if() function</indexterm>
+          <b>Purpose:</b> Tests an expression and returns a corresponding result depending on whether the result is
+          true, false, or <codeph>NULL</codeph>.
+          <p>
+            <b>Return type:</b> Same as the <codeph>ifTrue</codeph> argument value
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="ifnull">
+
+        <dt>
+          <codeph>ifnull(type a, type ifNull)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">isnull() function</indexterm>
+          <b>Purpose:</b> Alias for the <codeph>isnull()</codeph> function, with the same behavior. To simplify
+          porting SQL with vendor extensions to Impala.
+          <p conref="../shared/impala_common.xml#common/added_in_130"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="isfalse" rev="2.2.0">
+
+        <dt>
+          <codeph>isfalse(<varname>boolean</varname>)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">isfalse() function</indexterm>
+          <b>Purpose:</b> Tests if a Boolean expression is <codeph>false</codeph> or not.
+          Returns <codeph>true</codeph> if so.
+          If the argument is <codeph>NULL</codeph>, returns <codeph>false</codeph>.
+          Identical to <codeph>isnottrue()</codeph>, except it returns the opposite value for a <codeph>NULL</codeph> argument.
+          <p conref="../shared/impala_common.xml#common/return_type_boolean"/>
+          <p conref="../shared/impala_common.xml#common/added_in_220"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="isnotfalse" rev="2.2.0">
+
+        <dt>
+          <codeph>isnotfalse(<varname>boolean</varname>)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">isnotfalse() function</indexterm>
+          <b>Purpose:</b> Tests if a Boolean expression is not <codeph>false</codeph> (that is, either <codeph>true</codeph> or <codeph>NULL</codeph>).
+          Returns <codeph>true</codeph> if so.
+          If the argument is <codeph>NULL</codeph>, returns <codeph>true</codeph>.
+          Identical to <codeph>istrue()</codeph>, except it returns the opposite value for a <codeph>NULL</codeph> argument.
+          <p conref="../shared/impala_common.xml#common/return_type_boolean"/>
+          <p conref="../shared/impala_common.xml#common/for_compatibility_only"/>
+          <p conref="../shared/impala_common.xml#common/added_in_220"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="isnottrue" rev="2.2.0">
+
+        <dt>
+          <codeph>isnottrue(<varname>boolean</varname>)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">isnottrue() function</indexterm>
+          <b>Purpose:</b> Tests if a Boolean expression is not <codeph>true</codeph> (that is, either <codeph>false</codeph> or <codeph>NULL</codeph>).
+          Returns <codeph>true</codeph> if so.
+          If the argument is <codeph>NULL</codeph>, returns <codeph>true</codeph>.
+          Identical to <codeph>isfalse()</codeph>, except it returns the opposite value for a <codeph>NULL</codeph> argument.
+          <p conref="../shared/impala_common.xml#common/return_type_boolean"/>
+          <p conref="../shared/impala_common.xml#common/added_in_220"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="isnull">
+
+        <dt>
+          <codeph>isnull(type a, type ifNull)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">isnull() function</indexterm>
+          <b>Purpose:</b> Tests if an expression is <codeph>NULL</codeph>, and returns the expression result value
+          if not. If the first argument is <codeph>NULL</codeph>, returns the second argument.
+          <p>
+            <b>Compatibility notes:</b> Equivalent to the <codeph>nvl()</codeph> function from Oracle Database or
+            <codeph>ifnull()</codeph> from MySQL. The <codeph>nvl()</codeph> and <codeph>ifnull()</codeph>
+            functions are also available in Impala.
+          </p>
+          <p>
+            <b>Return type:</b> Same as the first argument value
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="istrue" rev="2.2.0">
+
+        <dt>
+          <codeph>istrue(<varname>boolean</varname>)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">istrue() function</indexterm>
+          <b>Purpose:</b> Tests if a Boolean expression is <codeph>true</codeph> or not.
+          Returns <codeph>true</codeph> if so.
+          If the argument is <codeph>NULL</codeph>, returns <codeph>false</codeph>.
+          Identical to <codeph>isnotfalse()</codeph>, except it returns the opposite value for a <codeph>NULL</codeph> argument.
+          <p conref="../shared/impala_common.xml#common/return_type_boolean"/>
+          <p conref="../shared/impala_common.xml#common/for_compatibility_only"/>
+          <p conref="../shared/impala_common.xml#common/added_in_220"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="nonnullvalue" rev="2.2.0">
+
+        <dt>
+          <codeph>nonnullvalue(<varname>expression</varname>)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">function</indexterm>
+          <b>Purpose:</b> Tests if an expression (of any type) is <codeph>NULL</codeph> or not.
+          Returns <codeph>false</codeph> if so.
+          The converse of <codeph>nullvalue()</codeph>.
+          <p conref="../shared/impala_common.xml#common/return_type_boolean"/>
+          <p conref="../shared/impala_common.xml#common/for_compatibility_only"/>
+          <p conref="../shared/impala_common.xml#common/added_in_220"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="nullif">
+
+        <dt>
+          <codeph>nullif(<varname>expr1</varname>,<varname>expr2</varname>)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">nullif() function</indexterm>
+          <b>Purpose:</b> Returns <codeph>NULL</codeph> if the two specified arguments are equal. If the specified
+          arguments are not equal, returns the value of <varname>expr1</varname>. The data types of the expressions
+          must be compatible, according to the conversion rules from <xref href="impala_datatypes.xml#datatypes"/>.
+          You cannot use an expression that evaluates to <codeph>NULL</codeph> for <varname>expr1</varname>; that
+          way, you can distinguish a return value of <codeph>NULL</codeph> from an argument value of
+          <codeph>NULL</codeph>, which would never match <varname>expr2</varname>.
+          <p>
+            <b>Usage notes:</b> This function is effectively shorthand for a <codeph>CASE</codeph> expression of
+            the form:
+          </p>
+<codeblock>CASE
+  WHEN <varname>expr1</varname> = <varname>expr2</varname> THEN NULL
+  ELSE <varname>expr1</varname>
+END</codeblock>
+          <p>
+            It is commonly used in division expressions, to produce a <codeph>NULL</codeph> result instead of a
+            divide-by-zero error when the divisor is equal to zero:
+          </p>
+<codeblock>select 1.0 / nullif(c1,0) as reciprocal from t1;</codeblock>
+          <p>
+            You might also use it for compatibility with other database systems that support the same
+            <codeph>NULLIF()</codeph> function.
+          </p>
+          <p conref="../shared/impala_common.xml#common/return_same_type"/>
+          <p conref="../shared/impala_common.xml#common/added_in_130"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="nullifzero">
+
+        <dt>
+          <codeph>nullifzero(<varname>numeric_expr</varname>)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">nullifzero() function</indexterm>
+          <b>Purpose:</b> Returns <codeph>NULL</codeph> if the numeric expression evaluates to 0, otherwise returns
+          the result of the expression.
+          <p>
+            <b>Usage notes:</b> Used to avoid error conditions such as divide-by-zero in numeric calculations.
+            Serves as shorthand for a more elaborate <codeph>CASE</codeph> expression, to simplify porting SQL with
+            vendor extensions to Impala.
+          </p>
+          <p conref="../shared/impala_common.xml#common/return_same_type"/>
+          <p conref="../shared/impala_common.xml#common/added_in_130"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="nullvalue" rev="2.2.0">
+
+        <dt>
+          <codeph>nullvalue(<varname>expression</varname>)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">function</indexterm>
+          <b>Purpose:</b> Tests if an expression (of any type) is <codeph>NULL</codeph> or not.
+          Returns <codeph>true</codeph> if so.
+          The converse of <codeph>nonnullvalue()</codeph>.
+          <p conref="../shared/impala_common.xml#common/return_type_boolean"/>
+          <p conref="../shared/impala_common.xml#common/for_compatibility_only"/>
+          <p conref="../shared/impala_common.xml#common/added_in_220"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="nvl" rev="1.1">
+
+        <dt>
+          <codeph>nvl(type a, type ifNull)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">nvl() function</indexterm>
+          <b>Purpose:</b> Alias for the <codeph>isnull()</codeph> function. Tests if an expression is
+          <codeph>NULL</codeph>, and returns the expression result value if not. If the first argument is
+          <codeph>NULL</codeph>, returns the second argument. Equivalent to the <codeph>nvl()</codeph> function
+          from Oracle Database or <codeph>ifnull()</codeph> from MySQL.
+          <p>
+            <b>Return type:</b> Same as the first argument value
+          </p>
+          <p conref="../shared/impala_common.xml#common/added_in_11"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="zeroifnull">
+
+        <dt>
+          <codeph>zeroifnull(<varname>numeric_expr</varname>)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">zeroifnull() function</indexterm>
+          <b>Purpose:</b> Returns 0 if the numeric expression evaluates to <codeph>NULL</codeph>, otherwise returns
+          the result of the expression.
+          <p>
+            <b>Usage notes:</b> Used to avoid unexpected results due to unexpected propagation of
+            <codeph>NULL</codeph> values in numeric calculations. Serves as shorthand for a more elaborate
+            <codeph>CASE</codeph> expression, to simplify porting SQL with vendor extensions to Impala.
+          </p>
+          <p conref="../shared/impala_common.xml#common/return_same_type"/>
+          <p conref="../shared/impala_common.xml#common/added_in_130"/>
+        </dd>
+
+      </dlentry>
+    </dl>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_config.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_config.xml b/docs/topics/impala_config.xml
new file mode 100644
index 0000000..7ea82e5
--- /dev/null
+++ b/docs/topics/impala_config.xml
@@ -0,0 +1,57 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="config">
+
+  <title>Managing Impala</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Configuring"/>
+      <data name="Category" value="JDBC"/>
+      <data name="Category" value="ODBC"/>
+      <data name="Category" value="Stub Pages"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      This section explains how to configure Impala to accept connections from applications that use popular
+      programming APIs:
+    </p>
+
+    <ul>
+      <li>
+        <xref href="impala_config_performance.xml#config_performance"/>
+      </li>
+
+      <li>
+        <xref href="impala_odbc.xml#impala_odbc"/>
+      </li>
+
+      <li>
+        <xref href="impala_jdbc.xml#impala_jdbc"/>
+      </li>
+    </ul>
+
+    <p>
+      This type of configuration is especially useful when using Impala in combination with Business Intelligence
+      tools, which use these standard interfaces to query different kinds of database and Big Data systems.
+    </p>
+
+    <p>
+      You can also configure these other aspects of Impala:
+    </p>
+
+    <ul>
+      <li>
+        <xref href="impala_security.xml#security"/>
+      </li>
+
+      <li>
+        <xref href="impala_config_options.xml#config_options"/>
+      </li>
+    </ul>
+  </conbody>
+</concept>


[21/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_ldap.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_ldap.xml b/docs/topics/impala_ldap.xml
new file mode 100644
index 0000000..f2ef523
--- /dev/null
+++ b/docs/topics/impala_ldap.xml
@@ -0,0 +1,354 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="ldap">
+
+  <title>Enabling LDAP Authentication for Impala</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="LDAP"/>
+      <data name="Category" value="Authentication"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Configuring"/>
+      <data name="Category" value="Starting and Stopping"/>
+      <data name="Category" value="Administrators"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+<!-- Similar discussion under 'Authentication' parent topic. Maybe do some conref'ing or linking upward. -->
+
+    <p> Authentication is the process of allowing only specified named users to
+      access the server (in this case, the Impala server). This feature is
+      crucial for any production deployment, to prevent misuse, tampering, or
+      excessive load on the server. Impala uses LDAP for authentication,
+      verifying the credentials of each user who connects through
+        <cmdname>impala-shell</cmdname>, Hue, a Business Intelligence tool, JDBC
+      or ODBC application, and so on. </p>
+
+    <note conref="../shared/impala_common.xml#common/authentication_vs_authorization"/>
+
+    <p outputclass="toc inpage"/>
+
+    <p>
+      An alternative form of authentication you can use is Kerberos, described in
+      <xref href="impala_kerberos.xml#kerberos"/>.
+    </p>
+  </conbody>
+
+  <concept id="ldap_prereqs">
+
+    <title>Requirements for Using Impala with LDAP</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Requirements"/>
+      <data name="Category" value="Planning"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p rev="1.4.0">
+        Authentication against LDAP servers is available in Impala 1.2.2 and higher. Impala 1.4.0 adds support for
+        secure LDAP authentication through SSL and TLS.
+      </p>
+
+      <p>
+        The Impala LDAP support lets you use Impala with systems such as Active Directory that use LDAP behind the
+        scenes.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="ldap_client_server">
+
+    <title>Client-Server Considerations for LDAP</title>
+
+    <conbody>
+
+      <p>
+        Only client-&gt;Impala connections can be authenticated by LDAP.
+      </p>
+
+      <p> You must use the Kerberos authentication mechanism for connections
+        between internal Impala components, such as between the
+          <cmdname>impalad</cmdname>, <cmdname>statestored</cmdname>, and
+          <cmdname>catalogd</cmdname> daemons. See <xref
+          href="impala_kerberos.xml#kerberos" /> on how to set up Kerberos for
+        Impala. </p>
+    </conbody>
+  </concept>
+
+  <concept id="ldap_config">
+
+    <title>Server-Side LDAP Setup</title>
+
+    <conbody>
+
+      <p>
+        These requirements apply on the server side when configuring and starting Impala:
+      </p>
+
+      <p>
+        To enable LDAP authentication, set the following startup options for <cmdname>impalad</cmdname>:
+      </p>
+
+      <ul>
+        <li>
+          <codeph>--enable_ldap_auth</codeph> enables LDAP-based authentication between the client and Impala.
+        </li>
+
+        <li rev="1.4.0">
+          <codeph>--ldap_uri</codeph> sets the URI of the LDAP server to use. Typically, the URI is prefixed with
+          <codeph>ldap://</codeph>. In Impala 1.4.0 and higher, you can specify secure SSL-based LDAP transport by
+          using the prefix <codeph>ldaps://</codeph>. The URI can optionally specify the port, for example:
+          <codeph>ldap://ldap_server.cloudera.com:389</codeph> or
+          <codeph>ldaps://ldap_server.cloudera.com:636</codeph>. (389 and 636 are the default ports for non-SSL and
+          SSL LDAP connections, respectively.)
+        </li>
+
+<!-- Some amount of this bullet could be conref'ed. Similar but not identical bullet occurs later under TLS. -->
+
+        <li rev="1.4.0">
+          For <codeph>ldaps://</codeph> connections secured by SSL,
+          <codeph>--ldap_ca_certificate="<varname>/path/to/certificate/pem</varname>"</codeph> specifies the
+          location of the certificate in standard <codeph>.PEM</codeph> format. Store this certificate on the local
+          filesystem, in a location that only the <codeph>impala</codeph> user and other trusted users can read.
+        </li>
+
+<!-- Per Henry: not for public consumption.
+<li>
+  If you need to provide a custom SASL configuration,
+  set <codeph>- -ldap_manual_config</codeph> to bypass all the automatic configuration.
+</li>
+-->
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="ldap_bind_strings">
+
+    <title>Support for Custom Bind Strings</title>
+
+    <conbody>
+
+      <p>
+        When Impala connects to LDAP it issues a bind call to the LDAP server to authenticate as the connected
+        user. Impala clients, including the Impala shell, provide the short name of the user to Impala. This is
+        necessary so that Impala can use Sentry for role-based access, which uses short names.
+      </p>
+
+      <p>
+        However, LDAP servers often require more complex, structured usernames for authentication. Impala supports
+        three ways of transforming the short name (for example, <codeph>'henry'</codeph>) to a more complicated
+        string. If necessary, specify one of the following configuration options when starting the
+        <cmdname>impalad</cmdname> daemon on each DataNode:
+      </p>
+
+      <ul>
+        <li>
+          <codeph>--ldap_domain</codeph>: Replaces the username with a string
+          <codeph><varname>username</varname>@<varname>ldap_domain</varname></codeph>.
+        </li>
+
+        <li>
+          <codeph>--ldap_baseDN</codeph>: Replaces the username with a <q>distinguished name</q> (DN) of the form:
+          <codeph>uid=<varname>userid</varname>,ldap_baseDN</codeph>. (This is equivalent to a Hive option).
+        </li>
+
+        <li>
+          <codeph>--ldap_bind_pattern</codeph>: This is the most general option, and replaces the username with the
+          string <varname>ldap_bind_pattern</varname> where all instances of the string <codeph>#UID</codeph> are
+          replaced with <varname>userid</varname>. For example, an <codeph>ldap_bind_pattern</codeph> of
+          <codeph>"user=#UID,OU=foo,CN=bar"</codeph> with a username of <codeph>henry</codeph> will construct a
+          bind name of <codeph>"user=henry,OU=foo,CN=bar"</codeph>.
+        </li>
+      </ul>
+
+      <p rev="CDH-26854">
+        For clusters not managed by Cloudera Manager,
+        specify the option on the <cmdname>impalad</cmdname> command line.
+        For clusters managed by Cloudera Manager 5.4.0 and higher,
+        search for the configuration field names <codeph>ldap_domain</codeph>,
+        <codeph>ldap_basedn</codeph>, or <codeph>ldap_bind_pattern</codeph>,
+        fill in and save the appropriate field values, and restart the Impala service.
+        Prior to Cloudera Manager 5.4.0, these values were filled in using the
+        <uicontrol>Impala Daemon Command Line Argument Advanced Configuration Snippet (Safety Valve)</uicontrol>
+        field.
+      </p>
+
+      <p>
+        These options are mutually exclusive; Impala does not start if more than one of these options is specified.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="ldap_security">
+
+    <title>Secure LDAP Connections</title>
+
+    <conbody>
+
+      <p>
+        To avoid sending credentials over the wire in cleartext, you must configure a secure connection between
+        both the client and Impala, and between Impala and the LDAP server. The secure connection could use SSL or
+        TLS.
+      </p>
+
+      <p>
+        <b>Secure LDAP connections through SSL:</b>
+      </p>
+
+      <p>
+        For SSL-enabled LDAP connections, specify a prefix of <codeph>ldaps://</codeph> instead of
+        <codeph>ldap://</codeph>. Also, the default port for SSL-enabled LDAP connections is 636 instead of 389.
+      </p>
+
+      <p rev="1.4.0">
+        <b>Secure LDAP connections through TLS:</b>
+      </p>
+
+      <p>
+        <xref href="http://en.wikipedia.org/wiki/Transport_Layer_Security" scope="external" format="html">TLS</xref>,
+        the successor to the SSL protocol, is supported by most modern LDAP servers. Unlike SSL connections, TLS
+        connections can be made on the same server port as non-TLS connections. To secure all connections using
+        TLS, specify the following flags as startup options to the <cmdname>impalad</cmdname> daemon:
+      </p>
+
+      <ul>
+        <li>
+          <codeph>--ldap_tls</codeph> tells Impala to start a TLS connection to the LDAP server, and to fail
+          authentication if it cannot be done.
+        </li>
+
+        <li rev="1.4.0">
+          <codeph>--ldap_ca_certificate="<varname>/path/to/certificate/pem</varname>"</codeph> specifies the
+          location of the certificate in standard <codeph>.PEM</codeph> format. Store this certificate on the local
+          filesystem, in a location that only the <codeph>impala</codeph> user and other trusted users can read.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="ldap_impala_shell">
+
+    <title>LDAP Authentication for impala-shell Interpreter</title>
+
+    <conbody>
+
+      <p>
+        To connect to Impala using LDAP authentication, you specify command-line options to the
+        <cmdname>impala-shell</cmdname> command interpreter and enter the password when prompted:
+      </p>
+
+      <ul>
+        <li>
+          <codeph>-l</codeph> enables LDAP authentication.
+        </li>
+
+        <li>
+          <codeph>-u</codeph> sets the user. Per Active Directory, the user is the short username, not the full
+          LDAP distinguished name. If your LDAP settings include a search base, use the
+          <codeph>--ldap_bind_pattern</codeph> on the <cmdname>impalad</cmdname> daemon to translate the short user
+          name from <cmdname>impala-shell</cmdname> automatically to the fully qualified name.
+<!--
+include that as part of the
+username, for example <codeph>username@example.com</codeph>.
+-->
+        </li>
+
+        <li>
+          <cmdname>impala-shell</cmdname> automatically prompts for the password.
+        </li>
+      </ul>
+
+      <p>
+        For the full list of available <cmdname>impala-shell</cmdname> options, see
+        <xref href="impala_shell_options.xml#shell_options"/>.
+      </p>
+
+      <p>
+        <b>LDAP authentication for JDBC applications:</b> See <xref href="impala_jdbc.xml#impala_jdbc"/> for the
+        format to use with the JDBC connection string for servers using LDAP authentication.
+      </p>
+    </conbody>
+  </concept>
+  <concept id="ldap_impala_hue">
+    <title>Enabling LDAP for Impala in Hue</title>
+    <prolog>
+      <metadata>
+        <data name="Category" value="Hue"/>
+      </metadata>
+    </prolog>
+    <conbody>
+      <section id="ldap_impala_hue_cm">
+        <title>Enabling LDAP for Impala in Hue Using Cloudera Manager</title>
+        <p>
+          <ol>
+            <li>Go to the Hue service.</li>
+            <li>Click the Configuration tab.</li>
+            <li>Select <menucascade><uicontrol>Scope</uicontrol><uicontrol>Hue
+                  Server</uicontrol></menucascade>.</li>
+            <li>Select
+              <menucascade><uicontrol>Category</uicontrol><uicontrol>Advanced</uicontrol></menucascade>.</li>
+            <li>Add the following properties to the <b>Hue Server Advanced
+                Configuration Snippet (Safety Valve) for
+                hue_safety_valve_server.ini</b>
+              property.<codeblock>[impala]
+auth_username=&lt;LDAP username of Hue user to be authenticated>
+auth_password=&lt;LDAP password of Hue user to be authenticated></codeblock></li>
+            <li>Click <b>Save Changes</b>.</li>
+          </ol>
+        </p>
+      </section>
+      <section id="ldap_impala_hue_cmdline">
+        <title>Enabling LDAP for Impala in Hue Using the Command Line</title>
+        <p>LDAP authentication for the Impala app in Hue can be enabled by
+          setting the following properties under the <codeph>[impala]</codeph>
+          section in <codeph>hue.ini</codeph>. <table id="ldap_impala_hue_configs">
+            <tgroup cols="2">
+              <colspec colname="1" colwidth="1*" />
+              <colspec colname="2" colwidth="2*" />
+              <tbody>
+                <row>
+                  <entry><codeph>auth_username</codeph></entry>
+                  <entry>LDAP username of Hue user to be authenticated.</entry>
+                </row>
+                <row>
+                  <entry><codeph>auth_password</codeph></entry>
+                  <entry>
+                    <p>LDAP password of Hue user to be authenticated.</p>
+                  </entry>
+                </row>
+              </tbody>
+            </tgroup>
+          </table>These login details are only used by Impala to authenticate to
+          LDAP. The Impala service trusts Hue to have already validated the user
+          being impersonated, rather than simply passing on the credentials.</p>
+      </section>
+    </conbody>
+  </concept>
+
+  <concept id="ldap_delegation">
+    <title>Enabling Impala Delegation for LDAP Users</title>
+    <conbody>
+      <p>
+        See <xref href="impala_delegation.xml#delegation"/> for details about the delegation feature
+        that lets certain users submit queries using the credentials of other users.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="ldap_restrictions">
+
+    <title>LDAP Restrictions for Impala</title>
+
+    <conbody>
+
+      <p>
+        The LDAP support is preliminary. It currently has only been tested against Active Directory.
+      </p>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_limit.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_limit.xml b/docs/topics/impala_limit.xml
new file mode 100644
index 0000000..ec12271
--- /dev/null
+++ b/docs/topics/impala_limit.xml
@@ -0,0 +1,151 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="limit">
+
+  <title>LIMIT Clause</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Reports"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      The <codeph>LIMIT</codeph> clause in a <codeph>SELECT</codeph> query sets a maximum number of rows for the
+      result set. Pre-selecting the maximum size of the result set helps Impala to optimize memory usage while
+      processing a distributed query.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>LIMIT <varname>constant_integer_expression</varname></codeblock>
+
+    <p>
+      The argument to the <codeph>LIMIT</codeph> clause must evaluate to a constant value. It can be a numeric
+      literal, or another kind of numeric expression involving operators, casts, and function return values. You
+      cannot refer to a column or use a subquery.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      This clause is useful in contexts such as:
+    </p>
+
+    <ul>
+      <li>
+        To return exactly N items from a top-N query, such as the 10 highest-rated items in a shopping category or
+        the 50 hostnames that refer the most traffic to a web site.
+      </li>
+
+      <li>
+        To demonstrate some sample values from a table or a particular query. (To display some arbitrary items, use
+        a query with no <codeph>ORDER BY</codeph> clause. An <codeph>ORDER BY</codeph> clause causes additional
+        memory and/or disk usage during the query.)
+      </li>
+
+      <li>
+        To keep queries from returning huge result sets by accident if a table is larger than expected, or a
+        <codeph>WHERE</codeph> clause matches more rows than expected.
+      </li>
+    </ul>
+
+    <p rev="1.2.1">
+      Originally, the value for the <codeph>LIMIT</codeph> clause had to be a numeric literal. In Impala 1.2.1 and
+      higher, it can be a numeric expression.
+    </p>
+
+    <p rev="obwl" conref="../shared/impala_common.xml#common/order_by_limit"/>
+
+    <p>
+      See <xref href="impala_order_by.xml#order_by"/> for details.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/limit_and_offset"/>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/subquery_no_limit"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      The following example shows how the <codeph>LIMIT</codeph> clause caps the size of the result set, with the
+      limit being applied after any other clauses such as <codeph>WHERE</codeph>.
+    </p>
+
+<codeblock>[localhost:21000] &gt; create database limits;
+[localhost:21000] &gt; use limits;
+[localhost:21000] &gt; create table numbers (x int);
+[localhost:21000] &gt; insert into numbers values (1), (3), (4), (5), (2);
+Inserted 5 rows in 1.34s
+[localhost:21000] &gt; select x from numbers limit 100;
++---+
+| x |
++---+
+| 1 |
+| 3 |
+| 4 |
+| 5 |
+| 2 |
++---+
+Returned 5 row(s) in 0.26s
+[localhost:21000] &gt; select x from numbers limit 3;
++---+
+| x |
++---+
+| 1 |
+| 3 |
+| 4 |
++---+
+Returned 3 row(s) in 0.27s
+[localhost:21000] &gt; select x from numbers where x &gt; 2 limit 2;
++---+
+| x |
++---+
+| 3 |
+| 4 |
++---+
+Returned 2 row(s) in 0.27s</codeblock>
+
+    <p>
+      For top-N and bottom-N queries, you use the <codeph>ORDER BY</codeph> and <codeph>LIMIT</codeph> clauses
+      together:
+    </p>
+
+<codeblock rev="obwl">[localhost:21000] &gt; select x as "Top 3" from numbers order by x desc limit 3;
++-------+
+| top 3 |
++-------+
+| 5     |
+| 4     |
+| 3     |
++-------+
+[localhost:21000] &gt; select x as "Bottom 3" from numbers order by x limit 3;
++----------+
+| bottom 3 |
++----------+
+| 1        |
+| 2        |
+| 3        |
++----------+
+</codeblock>
+
+    <p>
+      You can use constant values besides integer literals as the <codeph>LIMIT</codeph> argument:
+    </p>
+
+<codeblock>-- Other expressions that yield constant integer values work too.
+SELECT x FROM t1 LIMIT 1e6;                        -- Limit is one million.
+SELECT x FROM t1 LIMIT length('hello world');      -- Limit is 11.
+SELECT x FROM t1 LIMIT 2+2;                        -- Limit is 4.
+SELECT x FROM t1 LIMIT cast(truncate(9.9) AS INT); -- Limit is 9.
+</codeblock>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_lineage.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_lineage.xml b/docs/topics/impala_lineage.xml
new file mode 100644
index 0000000..677fc3d
--- /dev/null
+++ b/docs/topics/impala_lineage.xml
@@ -0,0 +1,113 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="lineage" rev="2.2.0">
+
+  <title>Viewing Lineage Information for Impala Data</title>
+  <titlealts audience="PDF"><navtitle>Viewing Lineage Info</navtitle></titlealts>
+  <prolog>
+
+    <metadata>
+
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Lineage"/>
+      <data name="Category" value="Governance"/>
+      <data name="Category" value="Data Management"/>
+      <data name="Category" value="Navigator"/>
+      <data name="Category" value="Administrators"/>
+
+    </metadata>
+
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.2.0">
+      <indexterm audience="Cloudera">lineage</indexterm>
+      <indexterm audience="Cloudera">column lineage</indexterm>
+      <term>Lineage</term> is a feature in the Cloudera Navigator data
+      management component that helps you track where data originated, and how
+      data propagates through the system through SQL statements such as
+        <codeph>SELECT</codeph>, <codeph>INSERT</codeph>, and <codeph>CREATE
+        TABLE AS SELECT</codeph>. Impala is covered by the Cloudera Navigator
+      lineage features in <keyword keyref="impala22_full"/> and higher. </p>
+
+    <p>
+      This type of tracking is important in high-security configurations, especially in highly regulated industries
+      such as healthcare, pharmaceuticals, financial services and intelligence. For such kinds of sensitive data, it is important to know all
+      the places in the system that contain that data or other data derived from it; to verify who has accessed
+      that data; and to be able to doublecheck that the data used to make a decision was processed correctly and
+      not tampered with.
+    </p>
+
+    <p>
+      You interact with this feature through <term>lineage diagrams</term> showing relationships between tables and
+      columns. For instructions about interpreting lineage diagrams, see
+      <xref audience="integrated" href="cn_iu_lineage.xml" /><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/cn_iu_lineage.html" scope="external" format="html"/>.
+    </p>
+
+    <section id="column_lineage">
+
+      <title>Column Lineage</title>
+
+      <p>
+        <term>Column lineage</term> tracks information in fine detail, at the level of
+        particular columns rather than entire tables.
+      </p>
+
+      <p>
+        For example, if you have a table with information derived from web logs, you might copy that data into
+        other tables as part of the ETL process. The ETL operations might involve transformations through
+        expressions and function calls, and rearranging the columns into more or fewer tables
+        (<term>normalizing</term> or <term>denormalizing</term> the data). Then for reporting, you might issue
+        queries against multiple tables and views. In this example, column lineage helps you determine that data
+        that entered the system as <codeph>RAW_LOGS.FIELD1</codeph> was then turned into
+        <codeph>WEBSITE_REPORTS.IP_ADDRESS</codeph> through an <codeph>INSERT ... SELECT</codeph> statement. Or,
+        conversely, you could start with a reporting query against a view, and trace the origin of the data in a
+        field such as <codeph>TOP_10_VISITORS.USER_ID</codeph> back to the underlying table and even further back
+        to the point where the data was first loaded into Impala.
+      </p>
+
+      <p>
+        When you have tables where you need to track or control access to sensitive information at the column
+        level, see <xref href="impala_authorization.xml#authorization"/> for how to implement column-level
+        security. You set up authorization using the Sentry framework, create views that refer to specific sets of
+        columns, and then assign authorization privileges to those views rather than the underlying tables.
+      </p>
+
+    </section>
+
+    <section id="lineage_data">
+
+      <title>Lineage Data for Impala</title>
+
+      <p>
+        The lineage feature is enabled by default. When lineage logging is enabled, the serialized column lineage
+        graph is computed for each query and stored in a specialized log file in JSON format.
+      </p>
+
+      <p>
+        Impala records queries in the lineage log if they complete successfully, or fail due to authorization
+        errors. For write operations such as <codeph>INSERT</codeph> and <codeph>CREATE TABLE AS SELECT</codeph>,
+        the statement is recorded in the lineage log only if it successfully completes. Therefore, the lineage
+        feature tracks data that was accessed by successful queries, or that was attempted to be accessed by
+        unsuccessful queries that were blocked due to authorization failure. These kinds of queries represent data
+        that really was accessed, or where the attempted access could represent malicious activity.
+      </p>
+
+      <p>
+        Impala does not record in the lineage log queries that fail due to syntax errors or that fail or are
+        cancelled before they reach the stage of requesting rows from the result set.
+      </p>
+
+      <p>
+        To enable or disable this feature on a system not managed by Cloudera Manager, set or remove the
+        <codeph>-lineage_event_log_dir</codeph> configuration option for the <cmdname>impalad</cmdname> daemon. For
+        information about turning the lineage feature on and off through Cloudera Manager, see
+        <xref audience="integrated" href="datamgmt_impala_lineage_log.xml"/><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/datamgmt_impala_lineage_log.html" scope="external" format="html"/>.
+      </p>
+
+    </section>
+
+  </conbody>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_literals.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_literals.xml b/docs/topics/impala_literals.xml
new file mode 100644
index 0000000..d84d84c
--- /dev/null
+++ b/docs/topics/impala_literals.xml
@@ -0,0 +1,384 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="literals">
+
+  <title>Literals</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">literals</indexterm>
+      Each of the Impala data types has corresponding notation for literal values of that type. You specify literal
+      values in SQL statements, such as in the <codeph>SELECT</codeph> list or <codeph>WHERE</codeph> clause of a
+      query, or as an argument to a function call. See <xref href="impala_datatypes.xml#datatypes"/> for a complete
+      list of types, ranges, and conversion rules.
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="numeric_literals">
+
+    <title>Numeric Literals</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">numeric literals</indexterm>
+        To write literals for the integer types (<codeph>TINYINT</codeph>, <codeph>SMALLINT</codeph>,
+        <codeph>INT</codeph>, and <codeph>BIGINT</codeph>), use a sequence of digits with optional leading zeros.
+      </p>
+
+      <p rev="1.4.0">
+        To write literals for the floating-point types (<codeph rev="1.4.0">DECIMAL</codeph>,
+        <codeph>FLOAT</codeph>, and <codeph>DOUBLE</codeph>), use a sequence of digits with an optional decimal
+        point (<codeph>.</codeph> character). To preserve accuracy during arithmetic expressions, Impala interprets
+        floating-point literals as the <codeph>DECIMAL</codeph> type with the smallest appropriate precision and
+        scale, until required by the context to convert the result to <codeph>FLOAT</codeph> or
+        <codeph>DOUBLE</codeph>.
+      </p>
+
+      <p>
+        Integer values are promoted to floating-point when necessary, based on the context.
+      </p>
+
+      <p>
+        You can also use exponential notation by including an <codeph>e</codeph> character. For example,
+        <codeph>1e6</codeph> is 1 times 10 to the power of 6 (1 million). A number in exponential notation is
+        always interpreted as floating-point.
+      </p>
+
+      <p rev="tk">
+        When Impala encounters a numeric literal, it considers the type to be the <q>smallest</q> that can
+        accurately represent the value. The type is promoted to larger or more accurate types if necessary, based
+        on subsequent parts of an expression.
+      </p>
+      <p>
+        For example, you can see by the types Impala defines for the following table columns
+        how it interprets the corresponding numeric literals:
+      </p>
+<codeblock>[localhost:21000] > create table ten as select 10 as x;
++-------------------+
+| summary           |
++-------------------+
+| Inserted 1 row(s) |
++-------------------+
+[localhost:21000] > desc ten;
++------+---------+---------+
+| name | type    | comment |
++------+---------+---------+
+| x    | tinyint |         |
++------+---------+---------+
+
+[localhost:21000] > create table four_k as select 4096 as x;
++-------------------+
+| summary           |
++-------------------+
+| Inserted 1 row(s) |
++-------------------+
+[localhost:21000] > desc four_k;
++------+----------+---------+
+| name | type     | comment |
++------+----------+---------+
+| x    | smallint |         |
++------+----------+---------+
+
+[localhost:21000] > create table one_point_five as select 1.5 as x;
++-------------------+
+| summary           |
++-------------------+
+| Inserted 1 row(s) |
++-------------------+
+[localhost:21000] > desc one_point_five;
++------+--------------+---------+
+| name | type         | comment |
++------+--------------+---------+
+| x    | decimal(2,1) |         |
++------+--------------+---------+
+
+[localhost:21000] > create table one_point_three_three_three as select 1.333 as x;
++-------------------+
+| summary           |
++-------------------+
+| Inserted 1 row(s) |
++-------------------+
+[localhost:21000] > desc one_point_three_three_three;
++------+--------------+---------+
+| name | type         | comment |
++------+--------------+---------+
+| x    | decimal(4,3) |         |
++------+--------------+---------+
+</codeblock>
+    </conbody>
+  </concept>
+
+  <concept id="string_literals">
+
+    <title>String Literals</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">string literals</indexterm>
+        String literals are quoted using either single or double quotation marks. You can use either kind of quotes
+        for string literals, even both kinds for different literals within the same statement.
+      </p>
+
+      <p rev="2.0.0">
+        Quoted literals are considered to be of type <codeph>STRING</codeph>. To use quoted literals in contexts
+        requiring a <codeph>CHAR</codeph> or <codeph>VARCHAR</codeph> value, <codeph>CAST()</codeph> the literal to
+        a <codeph>CHAR</codeph> or <codeph>VARCHAR</codeph> of the appropriate length.
+      </p>
+
+      <p>
+        <b>Escaping special characters:</b>
+      </p>
+
+      <p>
+        To encode special characters within a string literal, precede them with the backslash (<codeph>\</codeph>)
+        escape character:
+      </p>
+
+      <ul>
+        <li>
+          <codeph>\t</codeph> represents a tab.
+        </li>
+
+        <li>
+          <codeph>\n</codeph> represents a newline or linefeed. This might cause extra line breaks in
+          <cmdname>impala-shell</cmdname> output.
+        </li>
+
+        <li>
+          <codeph>\r</codeph> represents a carriage return. This might cause unusual formatting (making it appear
+          that some content is overwritten) in <cmdname>impala-shell</cmdname> output.
+        </li>
+
+        <li>
+          <codeph>\b</codeph> represents a backspace. This might cause unusual formatting (making it appear that
+          some content is overwritten) in <cmdname>impala-shell</cmdname> output.
+        </li>
+
+        <li>
+          <codeph>\0</codeph> represents an ASCII <codeph>nul</codeph> character (not the same as a SQL
+          <codeph>NULL</codeph>). This might not be visible in <cmdname>impala-shell</cmdname> output.
+        </li>
+
+        <li>
+          <codeph>\Z</codeph> represents a DOS end-of-file character. This might not be visible in
+          <cmdname>impala-shell</cmdname> output.
+        </li>
+
+        <li>
+          <codeph>\%</codeph> and <codeph>\_</codeph> can be used to escape wildcard characters within the string
+          passed to the <codeph>LIKE</codeph> operator.
+        </li>
+
+        <li>
+          <codeph>\</codeph> followed by 3 octal digits represents the ASCII code of a single character; for
+          example, <codeph>\101</codeph> is ASCII 65, the character <codeph>A</codeph>.
+        </li>
+
+        <li>
+          Use two consecutive backslashes (<codeph>\\</codeph>) to prevent the backslash from being interpreted as
+          an escape character.
+        </li>
+
+        <li>
+          Use the backslash to escape single or double quotation mark characters within a string literal, if the
+          literal is enclosed by the same type of quotation mark.
+        </li>
+
+        <li>
+          If the character following the <codeph>\</codeph> does not represent the start of a recognized escape
+          sequence, the character is passed through unchanged.
+        </li>
+      </ul>
+
+      <p>
+        <b>Quotes within quotes:</b>
+      </p>
+
+      <p>
+        To include a single quotation character within a string value, enclose the literal with either single or
+        double quotation marks, and optionally escape the single quote as a <codeph>\'</codeph> sequence. Earlier
+        releases required escaping a single quote inside double quotes. Continue using escape sequences in this
+        case if you also need to run your SQL code on older versions of Impala.
+      </p>
+
+      <p>
+        To include a double quotation character within a string value, enclose the literal with single quotation
+        marks, no escaping is necessary in this case. Or, enclose the literal with double quotation marks and
+        escape the double quote as a <codeph>\"</codeph> sequence.
+      </p>
+
+<codeblock>[localhost:21000] &gt; select "What\'s happening?" as single_within_double,
+                  &gt;        'I\'m not sure.' as single_within_single,
+                  &gt;        "Homer wrote \"The Iliad\"." as double_within_double,
+                  &gt;        'Homer also wrote "The Odyssey".' as double_within_single;
++----------------------+----------------------+--------------------------+---------------------------------+
+| single_within_double | single_within_single | double_within_double     | double_within_single            |
++----------------------+----------------------+--------------------------+---------------------------------+
+| What's happening?    | I'm not sure.        | Homer wrote "The Iliad". | Homer also wrote "The Odyssey". |
++----------------------+----------------------+--------------------------+---------------------------------+
+</codeblock>
+
+      <p>
+        <b>Field terminator character in CREATE TABLE:</b>
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/thorn"/>
+
+      <p>
+        <b>impala-shell considerations:</b>
+      </p>
+
+      <p>
+        When dealing with output that includes non-ASCII or non-printable characters such as linefeeds and
+        backspaces, use the <cmdname>impala-shell</cmdname> options to save to a file, turn off pretty printing, or
+        both rather than relying on how the output appears visually. See
+        <xref href="impala_shell_options.xml#shell_options"/> for a list of <cmdname>impala-shell</cmdname>
+        options.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="boolean_literals">
+
+    <title>Boolean Literals</title>
+
+    <conbody>
+
+      <p>
+        For <codeph>BOOLEAN</codeph> values, the literals are <codeph>TRUE</codeph> and <codeph>FALSE</codeph>,
+        with no quotation marks and case-insensitive.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>select true;
+select * from t1 where assertion = false;
+select case bool_col when true then 'yes' when false 'no' else 'null' end from t1;</codeblock>
+    </conbody>
+  </concept>
+
+  <concept id="timestamp_literals">
+
+    <title>Timestamp Literals</title>
+
+    <conbody>
+
+      <p conref="../shared/impala_common.xml#common/timestamp_conversions"/>
+
+      <p>
+        You can also use <codeph>INTERVAL</codeph> expressions to add or subtract from timestamp literal values,
+        such as <codeph>'1966-07-30' + INTERVAL 5 YEARS + INTERVAL 3 DAYS</codeph>. See
+        <xref href="impala_timestamp.xml#timestamp"/> for details.
+      </p>
+
+      <p>
+        Depending on your data pipeline, you might receive date and time data as text, in notation that does not
+        exactly match the format for Impala <codeph>TIMESTAMP</codeph> literals.
+        See <xref href="impala_datetime_functions.xml#datetime_functions"/> for functions that can convert
+        between a variety of string literals (including different field order, separators, and timezone notation)
+        and equivalent <codeph>TIMESTAMP</codeph> or numeric values.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="null">
+
+    <title>NULL</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">NULL</indexterm>
+        The notion of <codeph>NULL</codeph> values is familiar from all kinds of database systems, but each SQL
+        dialect can have its own behavior and restrictions on <codeph>NULL</codeph> values. For Big Data
+        processing, the precise semantics of <codeph>NULL</codeph> values are significant: any misunderstanding
+        could lead to inaccurate results or misformatted data, that could be time-consuming to correct for large
+        data sets.
+      </p>
+
+      <ul>
+        <li>
+          <codeph>NULL</codeph> is a different value than an empty string. The empty string is represented by a
+          string literal with nothing inside, <codeph>""</codeph> or <codeph>''</codeph>.
+        </li>
+
+        <li>
+          In a delimited text file, the <codeph>NULL</codeph> value is represented by the special token
+          <codeph>\N</codeph>.
+        </li>
+
+        <li>
+          When Impala inserts data into a partitioned table, and the value of one of the partitioning columns is
+          <codeph>NULL</codeph> or the empty string, the data is placed in a special partition that holds only
+          these two kinds of values. When these values are returned in a query, the result is <codeph>NULL</codeph>
+          whether the value was originally <codeph>NULL</codeph> or an empty string. This behavior is compatible
+          with the way Hive treats <codeph>NULL</codeph> values in partitioned tables. Hive does not allow empty
+          strings as partition keys, and it returns a string value such as
+          <codeph>__HIVE_DEFAULT_PARTITION__</codeph> instead of <codeph>NULL</codeph> when such values are
+          returned from a query. For example:
+<codeblock>create table t1 (i int) partitioned by (x int, y string);
+-- Select an INT column from another table, with all rows going into a special HDFS subdirectory
+-- named __HIVE_DEFAULT_PARTITION__. Depending on whether one or both of the partitioning keys
+-- are null, this special directory name occurs at different levels of the physical data directory
+-- for the table.
+insert into t1 partition(x=NULL, y=NULL) select c1 from some_other_table;
+insert into t1 partition(x, y=NULL) select c1, c2 from some_other_table;
+insert into t1 partition(x=NULL, y) select c1, c3  from some_other_table;</codeblock>
+        </li>
+
+        <li>
+          There is no <codeph>NOT NULL</codeph> clause when defining a column to prevent <codeph>NULL</codeph>
+          values in that column.
+        </li>
+
+        <li>
+          There is no <codeph>DEFAULT</codeph> clause to specify a non-<codeph>NULL</codeph> default value.
+        </li>
+
+        <li>
+          If an <codeph>INSERT</codeph> operation mentions some columns but not others, the unmentioned columns
+          contain <codeph>NULL</codeph> for all inserted rows.
+        </li>
+
+        <li rev="1.2.1">
+          <p conref="../shared/impala_common.xml#common/null_sorting_change"/>
+          <note>
+            <!-- To do: Probably a bunch of similar view-related restrictions like this that should be collected, reused, or cross-referenced under the Views topic. -->
+            Because the <codeph>NULLS FIRST</codeph> and <codeph>NULLS LAST</codeph> keywords are not currently
+            available in Hive queries, any views you create using those keywords will not be available through
+            Hive.
+          </note>
+        </li>
+
+        <li>
+          In all other contexts besides sorting with <codeph>ORDER BY</codeph>, comparing a <codeph>NULL</codeph>
+          to anything else returns <codeph>NULL</codeph>, making the comparison meaningless. For example,
+          <codeph>10 &gt; NULL</codeph> produces <codeph>NULL</codeph>, <codeph>10 &lt; NULL</codeph> also produces
+          <codeph>NULL</codeph>, <codeph>5 BETWEEN 1 AND NULL</codeph> produces <codeph>NULL</codeph>, and so on.
+        </li>
+      </ul>
+
+      <p>
+        Several built-in functions serve as shorthand for evaluating expressions and returning
+        <codeph>NULL</codeph>, 0, or some other substitution value depending on the expression result:
+        <codeph>ifnull()</codeph>, <codeph>isnull()</codeph>, <codeph>nvl()</codeph>, <codeph>nullif()</codeph>,
+        <codeph>nullifzero()</codeph>, and <codeph>zeroifnull()</codeph>. See
+        <xref href="impala_conditional_functions.xml#conditional_functions"/> for details.
+      </p>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_live_progress.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_live_progress.xml b/docs/topics/impala_live_progress.xml
new file mode 100644
index 0000000..f99e4b1
--- /dev/null
+++ b/docs/topics/impala_live_progress.xml
@@ -0,0 +1,86 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="2.3.0" id="live_progress">
+
+  <title>LIVE_PROGRESS Query Option (<keyword keyref="impala23"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>LIVE_PROGRESS</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Reports"/>
+      <data name="Category" value="impala-shell"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.3.0">
+      <indexterm audience="Cloudera">LIVE_PROGRESS query option</indexterm>
+      For queries submitted through the <cmdname>impala-shell</cmdname> command,
+      displays an interactive progress bar showing roughly what percentage of
+      processing has been completed. When the query finishes, the progress bar is erased
+      from the <cmdname>impala-shell</cmdname> console output.
+    </p>
+
+    <p>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_boolean"/>
+    <p conref="../shared/impala_common.xml#common/default_false_0"/>
+
+    <p conref="../shared/impala_common.xml#common/command_line_blurb"/>
+    <p>
+      You can enable this query option within <cmdname>impala-shell</cmdname>
+      by starting the shell with the <codeph>--live_progress</codeph>
+      command-line option.
+      You can still turn this setting off and on again within the shell through the
+      <codeph>SET</codeph> command.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+    <p conref="../shared/impala_common.xml#common/live_reporting_details"/>
+    <p>
+      For a more detailed way of tracking the progress of an interactive query through
+      all phases of processing, see <xref href="impala_live_summary.xml#live_summary"/>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+    <p>
+      Because the percentage complete figure is calculated using the number of
+      issued and completed <q>scan ranges</q>, which occur while reading the table
+      data, the progress bar might reach 100% before the query is entirely finished.
+      For example, the query might do work to perform aggregations after all the
+      table data has been read. If many of your queries fall into this category,
+      consider using the <codeph>LIVE_SUMMARY</codeph> option instead for
+      more granular progress reporting.
+    </p>
+    <p conref="../shared/impala_common.xml#common/impala_shell_progress_reports_compute_stats_caveat"/>
+    <p conref="../shared/impala_common.xml#common/impala_shell_progress_reports_shell_only_caveat"/>
+
+    <p conref="../shared/impala_common.xml#common/added_in_230"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock><![CDATA[[localhost:21000] > set live_progress=true;
+LIVE_PROGRESS set to true
+[localhost:21000] > select count(*) from customer;
++----------+
+| count(*) |
++----------+
+| 150000   |
++----------+
+[localhost:21000] > select count(*) from customer t1 cross join customer t2;
+[###################################                                   ] 50%
+[######################################################################] 100%
+
+]]>
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/live_progress_live_summary_asciinema"/>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_live_summary.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_live_summary.xml b/docs/topics/impala_live_summary.xml
new file mode 100644
index 0000000..9c7329b
--- /dev/null
+++ b/docs/topics/impala_live_summary.xml
@@ -0,0 +1,211 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="2.3.0" id="live_summary">
+
+  <title>LIVE_SUMMARY Query Option (<keyword keyref="impala23"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>LIVE_SUMMARY</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Reports"/>
+      <data name="Category" value="impala-shell"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.3.0">
+      <indexterm audience="Cloudera">LIVE_SUMMARY query option</indexterm>
+      For queries submitted through the <cmdname>impala-shell</cmdname> command,
+      displays the same output as the <codeph>SUMMARY</codeph> command,
+      with the measurements updated in real time as the query progresses.
+      When the query finishes, the final <codeph>SUMMARY</codeph> output remains
+      visible in the <cmdname>impala-shell</cmdname> console output.
+    </p>
+
+    <p>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_boolean"/>
+    <p conref="../shared/impala_common.xml#common/default_false_0"/>
+
+    <p conref="../shared/impala_common.xml#common/command_line_blurb"/>
+    <p>
+      You can enable this query option within <cmdname>impala-shell</cmdname>
+      by starting the shell with the <codeph>--live_summary</codeph>
+      command-line option.
+      You can still turn this setting off and on again within the shell through the
+      <codeph>SET</codeph> command.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+    <p>
+      The live summary output can be useful for evaluating long-running queries,
+      to evaluate which phase of execution takes up the most time, or if some hosts
+      take much longer than others for certain operations, dragging overall performance down.
+      By making the information available in real time, this feature lets you decide what
+      action to take even before you cancel a query that is taking much longer than normal.
+    </p>
+    <p>
+      For example, you might see the HDFS scan phase taking a long time, and therefore revisit
+      performance-related aspects of your schema design such as constructing a partitioned table,
+      switching to the Parquet file format, running the <codeph>COMPUTE STATS</codeph> statement
+      for the table, and so on.
+      Or you might see a wide variation between the average and maximum times for all hosts to
+      perform some phase of the query, and therefore investigate if one particular host
+      needed more memory or was experiencing a network problem.
+    </p>
+    <p conref="../shared/impala_common.xml#common/live_reporting_details"/>
+    <p>
+      For a simple and concise way of tracking the progress of an interactive query, see
+      <xref href="impala_live_progress.xml#live_progress"/>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+    <p conref="../shared/impala_common.xml#common/impala_shell_progress_reports_compute_stats_caveat"/>
+    <p conref="../shared/impala_common.xml#common/impala_shell_progress_reports_shell_only_caveat"/>
+
+    <p conref="../shared/impala_common.xml#common/added_in_230"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      The following example shows a series of <codeph>LIVE_SUMMARY</codeph> reports that
+      are displayed during the course of a query, showing how the numbers increase to
+      show the progress of different phases of the distributed query. When you do the same
+      in <cmdname>impala-shell</cmdname>, only a single report is displayed at any one time,
+      with each update overwriting the previous numbers.
+    </p>
+
+<codeblock><![CDATA[[localhost:21000] > set live_summary=true;
+LIVE_SUMMARY set to true
+[localhost:21000] > select count(*) from customer t1 cross join customer t2;
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| Operator            | #Hosts | Avg Time | Max Time | #Rows   | Est. #Rows | Peak Mem | Est. Peak Mem | Detail                |
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| 06:AGGREGATE        | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | -1 B          | FINALIZE              |
+| 05:EXCHANGE         | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | -1 B          | UNPARTITIONED         |
+| 03:AGGREGATE        | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | 10.00 MB      |                       |
+| 02:NESTED LOOP JOIN | 0      | 0ns      | 0ns      | 0       | 22.50B     | 0 B      | 0 B           | CROSS JOIN, BROADCAST |
+| |--04:EXCHANGE      | 0      | 0ns      | 0ns      | 0       | 150.00K    | 0 B      | 0 B           | BROADCAST             |
+| |  01:SCAN HDFS     | 1      | 503.57ms | 503.57ms | 150.00K | 150.00K    | 24.09 MB | 64.00 MB      | tpch.customer t2      |
+| 00:SCAN HDFS        | 0      | 0ns      | 0ns      | 0       | 150.00K    | 0 B      | 64.00 MB      | tpch.customer t1      |
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| Operator            | #Hosts | Avg Time | Max Time | #Rows   | Est. #Rows | Peak Mem | Est. Peak Mem | Detail                |
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| 06:AGGREGATE        | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | -1 B          | FINALIZE              |
+| 05:EXCHANGE         | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | -1 B          | UNPARTITIONED         |
+| 03:AGGREGATE        | 1      | 0ns      | 0ns      | 0       | 1          | 20.00 KB | 10.00 MB      |                       |
+| 02:NESTED LOOP JOIN | 1      | 17.62s   | 17.62s   | 81.14M  | 22.50B     | 3.23 MB  | 0 B           | CROSS JOIN, BROADCAST |
+| |--04:EXCHANGE      | 1      | 26.29ms  | 26.29ms  | 150.00K | 150.00K    | 0 B      | 0 B           | BROADCAST             |
+| |  01:SCAN HDFS     | 1      | 503.57ms | 503.57ms | 150.00K | 150.00K    | 24.09 MB | 64.00 MB      | tpch.customer t2      |
+| 00:SCAN HDFS        | 1      | 247.53ms | 247.53ms | 1.02K   | 150.00K    | 24.39 MB | 64.00 MB      | tpch.customer t1      |
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| Operator            | #Hosts | Avg Time | Max Time | #Rows   | Est. #Rows | Peak Mem | Est. Peak Mem | Detail                |
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| 06:AGGREGATE        | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | -1 B          | FINALIZE              |
+| 05:EXCHANGE         | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | -1 B          | UNPARTITIONED         |
+| 03:AGGREGATE        | 1      | 0ns      | 0ns      | 0       | 1          | 20.00 KB | 10.00 MB      |                       |
+| 02:NESTED LOOP JOIN | 1      | 61.85s   | 61.85s   | 283.43M | 22.50B     | 3.23 MB  | 0 B           | CROSS JOIN, BROADCAST |
+| |--04:EXCHANGE      | 1      | 26.29ms  | 26.29ms  | 150.00K | 150.00K    | 0 B      | 0 B           | BROADCAST             |
+| |  01:SCAN HDFS     | 1      | 503.57ms | 503.57ms | 150.00K | 150.00K    | 24.09 MB | 64.00 MB      | tpch.customer t2      |
+| 00:SCAN HDFS        | 1      | 247.59ms | 247.59ms | 2.05K   | 150.00K    | 24.39 MB | 64.00 MB      | tpch.customer t1      |
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+]]>
+</codeblock>
+
+<!-- Keeping this sample output that illustrates a couple of glitches in the LIVE_SUMMARY display, hidden, to help filing JIRAs. -->
+<codeblock audience="Cloudera"><![CDATA[[
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| Operator            | #Hosts | Avg Time | Max Time | #Rows   | Est. #Rows | Peak Mem | Est. Peak Mem | Detail                |
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| 06:AGGREGATE        | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | -1 B          | FINALIZE              |
+| 05:EXCHANGE         | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | -1 B          | UNPARTITIONED         |
+| 03:AGGREGATE        | 1      | 0ns      | 0ns      | 0       | 1          | 20.00 KB | 10.00 MB      |                       |
+| 02:NESTED LOOP JOIN | 1      | 91.34s   | 91.34s   | 419.48M | 22.50B     | 3.23 MB  | 0 B           | CROSS JOIN, BROADCAST |
+| |--04:EXCHANGE      | 1      | 26.29ms  | 26.29ms  | 150.00K | 150.00K    | 0 B      | 0 B           | BROADCAST             |
+| |  01:SCAN HDFS     | 1      | 503.57ms | 503.57ms | 150.00K | 150.00K    | 24.09 MB | 64.00 MB      | tpch.customer t2      |
+| 00:SCAN HDFS        | 1      | 247.63ms | 247.63ms | 3.07K   | 150.00K    | 24.39 MB | 64.00 MB      | tpch.customer t1      |
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| Operator            | #Hosts | Avg Time | Max Time | #Rows   | Est. #Rows | Peak Mem | Est. Peak Mem | Detail                |
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| 06:AGGREGATE        | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | -1 B          | FINALIZE              |
+| 05:EXCHANGE         | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | -1 B          | UNPARTITIONED         |
+| 03:AGGREGATE        | 1      | 0ns      | 0ns      | 0       | 1          | 20.00 KB | 10.00 MB      |                       |
+| 02:NESTED LOOP JOIN | 1      | 140.49s  | 140.49s  | 646.82M | 22.50B     | 3.23 MB  | 0 B           | CROSS JOIN, BROADCAST |
+| |--04:EXCHANGE      | 1      | 26.29ms  | 26.29ms  | 150.00K | 150.00K    | 0 B      | 0 B           | BROADCAST             |
+| |  01:SCAN HDFS     | 1      | 503.57ms | 503.57ms | 150.00K | 150.00K    | 24.09 MB | 64.00 MB      | tpch.customer t2      |
+| 00:SCAN HDFS        | 1      | 247.73ms | 247.73ms | 5.12K   | 150.00K    | 24.39 MB | 64.00 MB      | tpch.customer t1      |
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| Operator            | #Hosts | Avg Time | Max Time | #Rows   | Est. #Rows | Peak Mem | Est. Peak Mem | Detail                |
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| 06:AGGREGATE        | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | -1 B          | FINALIZE              |
+| 05:EXCHANGE         | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | -1 B          | UNPARTITIONED         |
+| 03:AGGREGATE        | 1      | 0ns      | 0ns      | 0       | 1          | 20.00 KB | 10.00 MB      |                       |
+| 02:NESTED LOOP JOIN | 1      | 228.96s  | 228.96s  | 1.06B   | 22.50B     | 3.23 MB  | 0 B           | CROSS JOIN, BROADCAST |
+| |--04:EXCHANGE      | 1      | 26.29ms  | 26.29ms  | 150.00K | 150.00K    | 0 B      | 0 B           | BROADCAST             |
+| |  01:SCAN HDFS     | 1      | 503.57ms | 503.57ms | 150.00K | 150.00K    | 24.09 MB | 64.00 MB      | tpch.customer t2      |
+| 00:SCAN HDFS        | 1      | 247.83ms | 247.83ms | 7.17K   | 150.00K    | 24.39 MB | 64.00 MB      | tpch.customer t1      |
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| Operator            | #Hosts | Avg Time | Max Time | #Rows   | Est. #Rows | Peak Mem | Est. Peak Mem | Detail                |
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| 06:AGGREGATE        | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | -1 B          | FINALIZE              |
+| 05:EXCHANGE         | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | -1 B          | UNPARTITIONED         |
+| 03:AGGREGATE        | 1      | 0ns      | 0ns      | 0       | 1          | 20.00 KB | 10.00 MB      |                       |
+| 02:NESTED LOOP JOIN | 1      | 563.11s  | 563.11s  | 2.59B   | 22.50B     | 3.23 MB  | 0 B           | CROSS JOIN, BROADCAST |
+| |--04:EXCHANGE      | 1      | 26.29ms  | 26.29ms  | 150.00K | 150.00K    | 0 B      | 0 B           | BROADCAST             |
+| |  01:SCAN HDFS     | 1      | 503.57ms | 503.57ms | 150.00K | 150.00K    | 24.09 MB | 64.00 MB      | tpch.customer t2      |
+| 00:SCAN HDFS        | 1      | 248.11ms | 248.11ms | 17.41K  | 150.00K    | 24.39 MB | 64.00 MB      | tpch.customer t1      |
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| Operator            | #Hosts | Avg Time | Max Time | #Rows   | Est. #Rows | Peak Mem | Est. Peak Mem | Detail                |
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| 06:AGGREGATE        | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | -1 B          | FINALIZE              |
+| 05:EXCHANGE         | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | -1 B          | UNPARTITIONED         |
+| 03:AGGREGATE        | 1      | 0ns      | 0ns      | 0       | 1          | 20.00 KB | 10.00 MB      |                       |
+| 02:NESTED LOOP JOIN | 1      | 985.71s  | 985.71s  | 4.54B   | 22.50B     | 3.23 MB  | 0 B           | CROSS JOIN, BROADCAST |
+| |--04:EXCHANGE      | 1      | 26.29ms  | 26.29ms  | 150.00K | 150.00K    | 0 B      | 0 B           | BROADCAST             |
+| |  01:SCAN HDFS     | 1      | 503.57ms | 503.57ms | 150.00K | 150.00K    | 24.09 MB | 64.00 MB      | tpch.customer t2      |
+| 00:SCAN HDFS        | 1      | 248.49ms | 248.49ms | 30.72K  | 150.00K    | 24.39 MB | 64.00 MB      | tpch.customer t1      |
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| Operator            | #Hosts | Avg Time | Max Time | #Rows   | Est. #Rows | Peak Mem | Est. Peak Mem | Detail                |
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| 06:AGGREGATE        | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | -1 B          | FINALIZE              |
+| 05:EXCHANGE         | 0      | 0ns      | 0ns      | 0       | 1          | 0 B      | -1 B          | UNPARTITIONED         |
+| 03:AGGREGATE        | 1      | 0ns      | 0ns      | 0       | 1          | 20.00 KB | 10.00 MB      |                       |
+| 02:NESTED LOOP JOIN | 1      | None     | None     | 5.42B   | 22.50B     | 3.23 MB  | 0 B           | CROSS JOIN, BROADCAST |
+| |--04:EXCHANGE      | 1      | 26.29ms  | 26.29ms  | 150.00K | 150.00K    | 0 B      | 0 B           | BROADCAST             |
+| |  01:SCAN HDFS     | 1      | 503.57ms | 503.57ms | 150.00K | 150.00K    | 24.09 MB | 64.00 MB      | tpch.customer t2      |
+| 00:SCAN HDFS        | 1      | 248.66ms | 248.66ms | 36.86K  | 150.00K    | 24.39 MB | 64.00 MB      | tpch.customer t1      |
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+
+[localhost:21000] > select count(*) from customer t1 cross join customer t2;
+Query: select count(*) from customer t1 cross join customer t2
+[####################################################################################################] 100%
++---------------------+--------+----------+----------+---------+------------+----------+---------------+-----------------------+
+| Operator            | #Hosts | Avg Time | Max Time | #Rows   | Est. #Rows | Peak Mem | Est. Peak Mem | Detail                |
+]]>
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/live_progress_live_summary_asciinema"/>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_load_data.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_load_data.xml b/docs/topics/impala_load_data.xml
new file mode 100644
index 0000000..0652914
--- /dev/null
+++ b/docs/topics/impala_load_data.xml
@@ -0,0 +1,256 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.1" id="load_data">
+
+  <title>LOAD DATA Statement</title>
+  <titlealts audience="PDF"><navtitle>LOAD DATA</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="ETL"/>
+      <data name="Category" value="Ingest"/>
+      <data name="Category" value="DML"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="HDFS"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="S3"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">LOAD DATA statement</indexterm>
+      The <codeph>LOAD DATA</codeph> statement streamlines the ETL process for an internal Impala table by moving a
+      data file or all the data files in a directory from an HDFS location into the Impala data directory for that
+      table.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>LOAD DATA INPATH '<varname>hdfs_file_or_directory_path</varname>' [OVERWRITE] INTO TABLE <varname>tablename</varname>
+  [PARTITION (<varname>partcol1</varname>=<varname>val1</varname>, <varname>partcol2</varname>=<varname>val2</varname> ...)]</codeblock>
+
+    <p>
+      When the <codeph>LOAD DATA</codeph> statement operates on a partitioned table,
+      it always operates on one partition at a time. Specify the <codeph>PARTITION</codeph> clauses
+      and list all the partition key columns, with a constant value specified for each.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/dml_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <ul>
+      <li>
+        The loaded data files are moved, not copied, into the Impala data directory.
+      </li>
+
+      <li>
+        You can specify the HDFS path of a single file to be moved, or the HDFS path of a directory to move all the
+        files inside that directory. You cannot specify any sort of wildcard to take only some of the files from a
+        directory. When loading a directory full of data files, keep all the data files at the top level, with no
+        nested directories underneath.
+      </li>
+
+      <li>
+        Currently, the Impala <codeph>LOAD DATA</codeph> statement only imports files from HDFS, not from the local
+        filesystem. It does not support the <codeph>LOCAL</codeph> keyword of the Hive <codeph>LOAD DATA</codeph>
+        statement. You must specify a path, not an <codeph>hdfs://</codeph> URI.
+      </li>
+
+      <li>
+        In the interest of speed, only limited error checking is done. If the loaded files have the wrong file
+        format, different columns than the destination table, or other kind of mismatch, Impala does not raise any
+        error for the <codeph>LOAD DATA</codeph> statement. Querying the table afterward could produce a runtime
+        error or unexpected results. Currently, the only checking the <codeph>LOAD DATA</codeph> statement does is
+        to avoid mixing together uncompressed and LZO-compressed text files in the same table.
+      </li>
+
+      <li>
+        When you specify an HDFS directory name as the <codeph>LOAD DATA</codeph> argument, any hidden files in
+        that directory (files whose names start with a <codeph>.</codeph>) are not moved to the Impala data
+        directory.
+      </li>
+
+      <li rev="2.5.0 IMPALA-2867">
+        The operation fails if the source directory contains any non-hidden directories.
+        Prior to <keyword keyref="impala25_full"/> if the source directory contained any subdirectory, even a hidden one such as
+        <filepath>_impala_insert_staging</filepath>, the <codeph>LOAD DATA</codeph> statement would fail.
+        In <keyword keyref="impala25_full"/> and higher, <codeph>LOAD DATA</codeph> ignores hidden subdirectories in the
+        source directory, and only fails if any of the subdirectories are non-hidden.
+      </li>
+
+      <li>
+        The loaded data files retain their original names in the new location, unless a name conflicts with an
+        existing data file, in which case the name of the new file is modified slightly to be unique. (The
+        name-mangling is a slight difference from the Hive <codeph>LOAD DATA</codeph> statement, which replaces
+        identically named files.)
+      </li>
+
+      <li>
+        By providing an easy way to transport files from known locations in HDFS into the Impala data directory
+        structure, the <codeph>LOAD DATA</codeph> statement lets you avoid memorizing the locations and layout of
+        HDFS directory tree containing the Impala databases and tables. (For a quick way to check the location of
+        the data files for an Impala table, issue the statement <codeph>DESCRIBE FORMATTED
+        <varname>table_name</varname></codeph>.)
+      </li>
+
+      <li>
+        The <codeph>PARTITION</codeph> clause is especially convenient for ingesting new data for a partitioned
+        table. As you receive new data for a time period, geographic region, or other division that corresponds to
+        one or more partitioning columns, you can load that data straight into the appropriate Impala data
+        directory, which might be nested several levels down if the table is partitioned by multiple columns. When
+        the table is partitioned, you must specify constant values for all the partitioning columns.
+      </li>
+    </ul>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+    <p rev="2.3.0">
+      Because Impala currently cannot create Parquet data files containing complex types
+      (<codeph>ARRAY</codeph>, <codeph>STRUCT</codeph>, and <codeph>MAP</codeph>), the
+      <codeph>LOAD DATA</codeph> statement is especially important when working with
+      tables containing complex type columns. You create the Parquet data files outside
+      Impala, then use either <codeph>LOAD DATA</codeph>, an external table, or HDFS-level
+      file operations followed by <codeph>REFRESH</codeph> to associate the data files with
+      the corresponding table.
+      See <xref href="impala_complex_types.xml#complex_types"/> for details about using complex types.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/sync_ddl_blurb"/>
+
+    <note conref="../shared/impala_common.xml#common/compute_stats_next"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      First, we use a trivial Python script to write different numbers of strings (one per line) into files stored
+      in the <codeph>cloudera</codeph> HDFS user account. (Substitute the path for your own HDFS user account when
+      doing <cmdname>hdfs dfs</cmdname> operations like these.)
+    </p>
+
+<codeblock>$ random_strings.py 1000 | hdfs dfs -put - /user/cloudera/thousand_strings.txt
+$ random_strings.py 100 | hdfs dfs -put - /user/cloudera/hundred_strings.txt
+$ random_strings.py 10 | hdfs dfs -put - /user/cloudera/ten_strings.txt</codeblock>
+
+    <p>
+      Next, we create a table and load an initial set of data into it. Remember, unless you specify a
+      <codeph>STORED AS</codeph> clause, Impala tables default to <codeph>TEXTFILE</codeph> format with Ctrl-A (hex
+      01) as the field delimiter. This example uses a single-column table, so the delimiter is not significant. For
+      large-scale ETL jobs, you would typically use binary format data files such as Parquet or Avro, and load them
+      into Impala tables that use the corresponding file format.
+    </p>
+
+<codeblock>[localhost:21000] &gt; create table t1 (s string);
+[localhost:21000] &gt; load data inpath '/user/cloudera/thousand_strings.txt' into table t1;
+Query finished, fetching results ...
++----------------------------------------------------------+
+| summary                                                  |
++----------------------------------------------------------+
+| Loaded 1 file(s). Total files in destination location: 1 |
++----------------------------------------------------------+
+Returned 1 row(s) in 0.61s
+[kilo2-202-961.cs1cloud.internal:21000] &gt; select count(*) from t1;
+Query finished, fetching results ...
++------+
+| _c0  |
++------+
+| 1000 |
++------+
+Returned 1 row(s) in 0.67s
+[localhost:21000] &gt; load data inpath '/user/cloudera/thousand_strings.txt' into table t1;
+ERROR: AnalysisException: INPATH location '/user/cloudera/thousand_strings.txt' does not exist. </codeblock>
+
+    <p>
+      As indicated by the message at the end of the previous example, the data file was moved from its original
+      location. The following example illustrates how the data file was moved into the Impala data directory for
+      the destination table, keeping its original filename:
+    </p>
+
+<codeblock>$ hdfs dfs -ls /user/hive/warehouse/load_data_testing.db/t1
+Found 1 items
+-rw-r--r--   1 cloudera cloudera      13926 2013-06-26 15:40 /user/hive/warehouse/load_data_testing.db/t1/thousand_strings.txt</codeblock>
+
+    <p>
+      The following example demonstrates the difference between the <codeph>INTO TABLE</codeph> and
+      <codeph>OVERWRITE TABLE</codeph> clauses. The table already contains 1000 rows. After issuing the
+      <codeph>LOAD DATA</codeph> statement with the <codeph>INTO TABLE</codeph> clause, the table contains 100 more
+      rows, for a total of 1100. After issuing the <codeph>LOAD DATA</codeph> statement with the <codeph>OVERWRITE
+      INTO TABLE</codeph> clause, the former contents are gone, and now the table only contains the 10 rows from
+      the just-loaded data file.
+    </p>
+
+<codeblock>[localhost:21000] &gt; load data inpath '/user/cloudera/hundred_strings.txt' into table t1;
+Query finished, fetching results ...
++----------------------------------------------------------+
+| summary                                                  |
++----------------------------------------------------------+
+| Loaded 1 file(s). Total files in destination location: 2 |
++----------------------------------------------------------+
+Returned 1 row(s) in 0.24s
+[localhost:21000] &gt; select count(*) from t1;
+Query finished, fetching results ...
++------+
+| _c0  |
++------+
+| 1100 |
++------+
+Returned 1 row(s) in 0.55s
+[localhost:21000] &gt; load data inpath '/user/cloudera/ten_strings.txt' overwrite into table t1;
+Query finished, fetching results ...
++----------------------------------------------------------+
+| summary                                                  |
++----------------------------------------------------------+
+| Loaded 1 file(s). Total files in destination location: 1 |
++----------------------------------------------------------+
+Returned 1 row(s) in 0.26s
+[localhost:21000] &gt; select count(*) from t1;
+Query finished, fetching results ...
++-----+
+| _c0 |
++-----+
+| 10  |
++-----+
+Returned 1 row(s) in 0.62s</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/s3_blurb"/>
+    <p conref="../shared/impala_common.xml#common/s3_dml"/>
+    <p conref="../shared/impala_common.xml#common/s3_dml_performance"/>
+    <p>See <xref href="../topics/impala_s3.xml#s3"/> for details about reading and writing S3 data with Impala.</p>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+    <p rev="CDH-19187">
+      The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+      typically the <codeph>impala</codeph> user, must have read and write
+      permissions for the files in the source directory, and write
+      permission for the destination directory.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/kudu_blurb"/>
+    <p conref="../shared/impala_common.xml#common/kudu_no_load_data"/>
+
+    <p conref="../shared/impala_common.xml#common/hbase_blurb"/>
+    <p conref="../shared/impala_common.xml#common/hbase_no_load_data"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      The <codeph>LOAD DATA</codeph> statement is an alternative to the
+      <codeph><xref href="impala_insert.xml#insert">INSERT</xref></codeph> statement.
+      Use <codeph>LOAD DATA</codeph>
+      when you have the data files in HDFS but outside of any Impala table.
+    </p>
+    <p>
+      The <codeph>LOAD DATA</codeph> statement is also an alternative
+      to the <codeph>CREATE EXTERNAL TABLE</codeph> statement. Use
+      <codeph>LOAD DATA</codeph> when it is appropriate to move the
+      data files under Impala control rather than querying them
+      from their original location. See <xref href="impala_tables.xml#external_tables"/>
+      for information about working with external tables.
+    </p>
+  </conbody>
+</concept>


[10/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_resource_management.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_resource_management.xml b/docs/topics/impala_resource_management.xml
new file mode 100644
index 0000000..abc2e3d
--- /dev/null
+++ b/docs/topics/impala_resource_management.xml
@@ -0,0 +1,338 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.2" id="resource_management">
+
+  <title>Resource Management for Impala</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="YARN"/>
+      <data name="Category" value="Resource Management"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <note conref="../shared/impala_common.xml#common/impala_llama_obsolete"/>
+
+    <p>
+      You can limit the CPU and memory resources used by Impala, to manage and prioritize workloads on clusters
+      that run jobs from many Hadoop components.
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept audience="Cloudera" id="llama">
+  <!-- Hiding the whole concept now that Llama is desupported. -->
+
+    <title>The Llama Daemon</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Llama"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+<!-- Copied from http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh_ig_install_llama.html - turn into a conref in both places. -->
+
+      <p>
+        Llama is a system that mediates resource management between Impala and Hadoop YARN. Llama enables
+        Impala to reserve, use, and release resource allocations in a Hadoop cluster. Llama is only required if
+        resource management is enabled in Impala.
+      </p>
+
+      <p id="p_b2y_msl_jp">
+        By default, YARN allocates resources bit-by-bit as needed by MapReduce jobs. Impala needs all resources
+        available at the same time, so that intermediate results can be exchanged between cluster nodes, and
+        queries do not stall partway through waiting for new resources to be allocated. Llama is the intermediary
+        process that ensures all requested resources are available before each Impala query actually begins.
+      </p>
+
+<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh_ig_llama_installation.html -->
+
+<!-- Highly likely to be removed per https://jira.cloudera.com/browse/CDH-21352,
+     so commenting out for now. Feels like there should be some 'installing llama via CM' topic
+     even if it says not to worry.
+
+      <p>
+        For Llama installation instructions, see
+        <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_llama_installation.html" scope="external" format="html">Llama
+        Installation</xref>.
+      </p>
+-->
+
+<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Managing-Clusters/cm_mc_impala_service.html -->
+
+      <p>
+        For management through Cloudera Manager, see
+        <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/admin_llama.html" scope="external" format="html">The
+        Impala Llama ApplicationMaster</xref>.
+      </p>
+    </conbody>
+  </concept>
+
+<!-- Hiding more content per MJ feedback.
+  <concept id="rm_resource_estimates">
+
+    <title>Controlling Resource Estimation Behavior</title>
+
+    <conbody>
+
+      <p>
+        By default, Impala consults the table statistics and column statistics for each table in a query, and uses
+        those figures to construct estimates of needed resources for each query. See
+        <xref href="impala_compute_stats.xml#compute_stats"/> for the statement to collect table and column
+        statistics for a table.
+      </p>
+
+      <p>
+        In <keyword keyref="impala25_full"/> and higher, the preferred way to avoid overcommitting memory in a high-concurrency
+        or multitenant scenario is to use Impala admission control together with dynamic resource pools.
+        You can specify a <uicontrol>Default Query Memory Limit</uicontrol> setting, with a different value for each
+        pool, and Impala uses that value to calculate how many queries can safely run within a specified
+        cluster-wide aggregate memory size.
+        See <xref href="impala_admission.xml#admission_control"/> for details.
+      </p>
+
+      <p>
+        To avoid problems with inaccurate or missing statistics, which can lead to inaccurate estimates of resource
+        consumption, Impala allows you to set default estimates for CPU and memory resource consumption.
+        When the query is complete, those resources are returned to YARN as
+        normal. To enable this feature, use the command-line option <codeph>-rm_always_use_defaults</codeph> when
+        starting <cmdname>impalad</cmdname>, and optionally
+        <codeph>-rm_default_memory=<varname>size</varname></codeph> and <codeph>-rm_default_cpu_cores</codeph>.
+        See <xref href="impala_resource_management.xml#rm_options"/> for details about each option.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="rm_checking">
+
+    <title>Checking Resource Estimates and Actual Usage</title>
+
+    <conbody>
+
+      <p>
+        To make resource usage easier to verify, the output of the <codeph>EXPLAIN</codeph> SQL statement now
+        includes information about estimated memory usage, whether table and column statistics are available for
+        each table, and the number of virtual cores that a query will use. You can get this information through the
+        <codeph>EXPLAIN</codeph> statement without actually running the query. The extra information requires
+        setting the query option <codeph>EXPLAIN_LEVEL=verbose</codeph>; see
+        <xref href="impala_explain.xml#explain"/> for details. The same extended information is shown at the start
+        of the output from the <codeph>PROFILE</codeph> statement in <cmdname>impala-shell</cmdname>. The detailed
+        profile information is only available after running the query. You can take appropriate actions (gathering
+        statistics, adjusting query options) if you find that queries fail or run with suboptimal performance when
+        resource management is enabled.
+      </p>
+    </conbody>
+  </concept>
+-->
+
+  <concept id="rm_enforcement">
+
+    <title>How Resource Limits Are Enforced</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Concepts"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <ul>
+        <li>
+          If Cloudera Manager Static Partitioning is used, it creates a cgroup in which Impala runs.
+          This cgroup limits CPU, network, and IO according to the static partitioning policy.
+        </li>
+
+        <li>
+          Limits on memory usage are enforced by Impala's process memory limit (the <codeph>MEM_LIMIT</codeph>
+          query option setting). The admission control feature checks this setting to decide how many queries
+          can be safely run at the same time. Then the Impala daemon enforces the limit by activating the
+          spill-to-disk mechanism when necessary, or cancelling a query altogether if the limit is exceeded at runtime.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+<!--
+  <concept id="rm_enable">
+
+    <title>Enabling Resource Management for Impala</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Configuring"/>
+      <data name="Category" value="Starting and Stopping"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        To enable resource management for Impala, first you <xref href="#rm_cdh_prereqs">set up the YARN
+        service for your CDH cluster</xref>. Then you <xref href="#rm_options">add startup options and customize
+        resource management settings</xref> for the Impala services.
+      </p>
+    </conbody>
+
+    <concept id="rm_cdh_prereqs">
+
+      <title>Required CDH Setup for Resource Management with Impala</title>
+
+      <conbody>
+
+        <p>
+          YARN is the general-purpose service that manages resources for many Hadoop components within a CDH
+          cluster.
+        </p>
+
+        <p>
+          For information about setting up the YARN service, see the instructions for
+          <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_yarn_service.html" scope="external" format="html">Cloudera
+          Manager</xref>.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept id="rm_options">
+
+      <title>impalad Startup Options for Resource Management</title>
+
+      <conbody>
+
+        <p id="resource_management_impalad_options">
+          The following startup options for <cmdname>impalad</cmdname> enable resource management and customize its
+          parameters for your cluster configuration:
+          <ul>
+            <li>
+              <codeph>-enable_rm</codeph>: Whether to enable resource management or not, either
+              <codeph>true</codeph> or <codeph>false</codeph>. The default is <codeph>false</codeph>. None of the
+              other resource management options have any effect unless <codeph>-enable_rm</codeph> is turned on.
+            </li>
+
+            <li>
+              <codeph>-cgroup_hierarchy_path</codeph>: Path where YARN will create cgroups for granted
+              resources. Impala assumes that the cgroup for an allocated container is created in the path
+              '<varname>cgroup_hierarchy_path</varname> + <varname>container_id</varname>'.
+            </li>
+
+            <li rev="1.4.0">
+              <codeph>-rm_always_use_defaults</codeph>: If this Boolean option is enabled, Impala ignores computed
+              estimates and always obtains the default memory and CPU allocation settings at the start of the
+              query. These default estimates are approximately 2 CPUs and 4 GB of memory, possibly varying slightly
+              depending on cluster size, workload, and so on. <ph rev="upstream">Cloudera</ph> recommends enabling
+              <codeph>-rm_always_use_defaults</codeph> whenever resource management is used, and relying on these
+              default values (that is, leaving out the two following options).
+            </li>
+
+            <li rev="1.4.0">
+              <codeph>-rm_default_memory=<varname>size</varname></codeph>: Optionally sets the default estimate for
+              memory usage for each query. You can use suffixes such as M and G for megabytes and gigabytes, the
+              same as with the <xref href="impala_mem_limit.xml#mem_limit">MEM_LIMIT</xref> query option. Only has
+              an effect when <codeph>-rm_always_use_defaults</codeph> is also enabled.
+            </li>
+
+            <li rev="1.4.0">
+              <codeph>-rm_default_cpu_cores</codeph>: Optionally sets the default estimate for number of virtual
+              CPU cores for each query. Only has an effect when <codeph>-rm_always_use_defaults</codeph> is also
+              enabled.
+            </li>
+          </ul>
+        </p>
+
+      </conbody>
+    </concept>
+-->
+
+    <concept id="rm_query_options">
+
+      <title>impala-shell Query Options for Resource Management</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala Query Options"/>
+    </metadata>
+  </prolog>
+
+      <conbody>
+
+        <p>
+          Before issuing SQL statements through the <cmdname>impala-shell</cmdname> interpreter, you can use the
+          <codeph>SET</codeph> command to configure the following parameters related to resource management:
+        </p>
+
+        <ul id="ul_nzt_twf_jp">
+          <li>
+            <xref href="impala_explain_level.xml#explain_level"/>
+          </li>
+
+          <li>
+            <xref href="impala_mem_limit.xml#mem_limit"/>
+          </li>
+
+<!-- Not supported in CDH 5.5 / Impala 2.3 and up.
+          <li>
+            <xref href="impala_reservation_request_timeout.xml#reservation_request_timeout"/>
+          </li>
+
+          <li>
+            <xref href="impala_v_cpu_cores.xml#v_cpu_cores"/>
+          </li>
+-->
+        </ul>
+      </conbody>
+    </concept>
+
+<!-- Parent topic is going away, so former subtopic is hoisted up a level.
+  </concept>
+-->
+
+  <concept id="rm_limitations">
+
+    <title>Limitations of Resource Management for Impala</title>
+
+    <conbody>
+
+<!-- Conditionalizing some content here with audience="Cloudera" because there are already some XML comments
+     inside the list, so not practical to enclose the whole thing in XML comments. -->
+
+      <p audience="Cloudera">
+        Currently, Impala in CDH 5 has the following limitations for resource management of Impala queries:
+      </p>
+
+      <ul audience="Cloudera">
+        <li>
+          Table statistics are required, and column statistics are highly valuable, for Impala to produce accurate
+          estimates of how much memory to request from YARN. See
+          <xref href="impala_perf_stats.xml#perf_table_stats"/> and
+          <xref href="impala_perf_stats.xml#perf_column_stats"/> for instructions on gathering both kinds of
+          statistics, and <xref href="impala_explain.xml#explain"/> for the extended <codeph>EXPLAIN</codeph>
+          output where you can check that statistics are available for a specific table and set of columns.
+        </li>
+
+        <li>
+          If the Impala estimate of required memory is lower than is actually required for a query, Impala
+          dynamically expands the amount of requested memory.
+<!--          Impala will cancel the query when it exceeds the requested memory size. -->
+          Queries might still be cancelled if the reservation expansion fails, for example if there are
+          insufficient remaining resources for that pool, or the expansion request takes long enough that it
+          exceeds the query timeout interval, or because of YARN preemption.
+<!--          This could happen in some cases with complex queries, even when table and column statistics are available. -->
+          You can see the actual memory usage after a failed query by issuing a <codeph>PROFILE</codeph> command in
+          <cmdname>impala-shell</cmdname>. Specify a larger memory figure with the <codeph>MEM_LIMIT</codeph>
+          query option and re-try the query.
+        </li>
+      </ul>
+
+      <p rev="2.0.0">
+        The <codeph>MEM_LIMIT</codeph> query option, and the other resource-related query options, are settable
+        through the ODBC or JDBC interfaces in Impala 2.0 and higher. This is a former limitation that is now
+        lifted.
+      </p>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_revoke.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_revoke.xml b/docs/topics/impala_revoke.xml
new file mode 100644
index 0000000..9ab81fe
--- /dev/null
+++ b/docs/topics/impala_revoke.xml
@@ -0,0 +1,101 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="2.0.0" id="revoke">
+
+  <title>REVOKE Statement (<keyword keyref="impala20"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>REVOKE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="Sentry"/>
+      <data name="Category" value="Roles"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+      <!-- Consider whether to go deeper into categories like Security for the Sentry-related statements. -->
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.0.0">
+      <indexterm audience="Cloudera">REVOKE statement</indexterm>
+<!-- Copied from Sentry docs. Turn into conref. I did some rewording for clarity. -->
+      The <codeph>REVOKE</codeph> statement revokes roles or privileges on a specified object from groups. Only
+      Sentry administrative users can revoke the role from a group. The revocation has a cascading effect. For
+      example, revoking the <codeph>ALL</codeph> privilege on a database also revokes the same privilege for all
+      the tables in that database.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock rev="2.3.0 collevelauth">REVOKE ROLE <varname>role_name</varname> FROM GROUP <varname>group_name</varname>
+
+REVOKE <varname>privilege</varname> ON <varname>object_type</varname> <varname>object_name</varname>
+  FROM [ROLE] <varname>role_name</varname>
+
+<ph rev="2.3.0">privilege ::= SELECT | SELECT(<varname>column_name</varname>) | INSERT | ALL</ph>
+object_type ::= TABLE | DATABASE | SERVER | URI
+</codeblock>
+
+    <p>
+      Typically, the object name is an identifier. For URIs, it is a string literal.
+    </p>
+
+    <p rev="2.3.0 collevelauth">
+      The ability to grant or revoke <codeph>SELECT</codeph> privilege on specific columns is available
+      in <keyword keyref="impala23_full"/> and higher. See
+      <xref audience="integrated" href="sg_hive_sql.xml#concept_c2q_4qx_p4/col_level_auth_sentry"/><xref audience="standalone" href="https://www.cloudera.com/documentation/enterprise/latest/topics/sg_hive_sql.html" format="html" scope="external"/>
+      for details.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/privileges_blurb"/>
+
+    <p>
+      Only administrative users (those with <codeph>ALL</codeph> privileges on the server, defined in the Sentry
+      policy file) can use this statement.
+    </p>
+
+<!-- Turn compatibility info into a conref or series of conrefs. (In both GRANT and REVOKE.) -->
+
+    <p conref="../shared/impala_common.xml#common/compatibility_blurb"/>
+
+    <p>
+      <ul>
+        <li>
+          The Impala <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements are available in CDH 5.2 and
+          higher.
+        </li>
+
+        <li>
+          In <ph rev="upstream">CDH 5.1</ph> and higher, Impala makes use of any roles and privileges specified by the
+          <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements in Hive, when your system is configured to
+          use the Sentry service instead of the file-based policy mechanism.
+        </li>
+
+        <li>
+          The Impala <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements do not require the
+          <codeph>ROLE</codeph> keyword to be repeated before each role name, unlike the equivalent Hive
+          statements.
+        </li>
+
+        <li conref="../shared/impala_common.xml#common/grant_revoke_single"/>
+      </ul>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_authorization.xml#authorization"/>, <xref href="impala_grant.xml#grant"/>
+      <xref href="impala_create_role.xml#create_role"/>, <xref href="impala_drop_role.xml#drop_role"/>,
+      <xref href="impala_show.xml#show"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_rm_initial_mem.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_rm_initial_mem.xml b/docs/topics/impala_rm_initial_mem.xml
new file mode 100644
index 0000000..cfd77cf
--- /dev/null
+++ b/docs/topics/impala_rm_initial_mem.xml
@@ -0,0 +1,29 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="rm_initial_mem" rev="2.5.0">
+
+  <title>RM_INITIAL_MEM Query Option</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.5.0">
+      <indexterm audience="Cloudera">RM_INITIAL_MEM query option</indexterm>
+    </p>
+
+    <p>
+      <b>Type:</b>
+    </p>
+
+    <p>
+      <b>Default:</b>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_runtime_bloom_filter_size.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_runtime_bloom_filter_size.xml b/docs/topics/impala_runtime_bloom_filter_size.xml
new file mode 100644
index 0000000..8fdc0c3
--- /dev/null
+++ b/docs/topics/impala_runtime_bloom_filter_size.xml
@@ -0,0 +1,93 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="runtime_bloom_filter_size" rev="2.5.0">
+
+  <title>RUNTIME_BLOOM_FILTER_SIZE Query Option (<keyword keyref="impala25"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>RUNTIME_BLOOM_FILTER_SIZE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.5.0">
+      <indexterm audience="Cloudera">RUNTIME_BLOOM_FILTER_SIZE query option</indexterm>
+      Size (in bytes) of Bloom filter data structure used by the runtime filtering
+      feature.
+    </p>
+
+    <note type="important">
+      <p rev="2.6.0 CDH-41184 IMPALA-3007">
+        In <keyword keyref="impala26_full"/> and higher, this query option only applies as a fallback, when statistics
+        are not available. By default, Impala estimates the optimal size of the Bloom filter structure
+        regardless of the setting for this option. (This is a change from the original behavior in
+        <keyword keyref="impala25_full"/>.)
+      </p>
+      <p rev="2.6.0 CDH-41184 IMPALA-3007">
+        In <keyword keyref="impala26_full"/> and higher, when the value of this query option is used for query planning,
+        it is constrained by the minimum and maximum sizes specified by the
+        <codeph>RUNTIME_FILTER_MIN_SIZE</codeph> and <codeph>RUNTIME_FILTER_MAX_SIZE</codeph> query options.
+        The filter size is adjusted upward or downward if necessary to fit within the minimum/maximum range.
+      </p>
+    </note>
+
+    <p conref="../shared/impala_common.xml#common/type_integer"/>
+
+    <p>
+      <b>Default:</b> 1048576 (1 MB)
+    </p>
+
+    <p>
+      <b>Maximum:</b> 16 MB
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/added_in_250"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      This setting affects optimizations for large and complex queries, such
+      as dynamic partition pruning for partitioned tables, and join optimization
+      for queries that join large tables.
+      Larger filters are more effective at handling
+      higher cardinality input sets, but consume more memory per filter.
+      <!--
+      See <xref href="impala_runtime_filtering.xml#runtime_filtering"/> for details about when to change
+      the setting for this query option.
+      -->
+    </p>
+
+    <p>
+      If your query filters on high-cardinality columns (for example, millions of different values)
+      and you do not get the expected speedup from the runtime filtering mechanism, consider
+      doing some benchmarks with a higher value for <codeph>RUNTIME_BLOOM_FILTER_SIZE</codeph>.
+      The extra memory devoted to the Bloom filter data structures can help make the filtering
+      more accurate.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/runtime_filtering_option_caveat"/>
+
+    <p>
+      Because the effectiveness of this setting depends so much on query characteristics and data distribution,
+      you typically only use it for specific queries that need some extra tuning, and the ideal value depends
+      on the query. Consider setting this query option immediately before the expensive query and
+      unsetting it immediately afterward.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_runtime_filtering.xml"/>,
+      <!-- , <xref href="impala_partitioning.xml#dynamic_partition_pruning"/> -->
+      <xref href="impala_runtime_filter_mode.xml#runtime_filter_mode"/>,
+      <xref href="impala_runtime_filter_min_size.xml#runtime_filter_min_size"/>,
+      <xref href="impala_runtime_filter_max_size.xml#runtime_filter_max_size"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_runtime_filter_max_size.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_runtime_filter_max_size.xml b/docs/topics/impala_runtime_filter_max_size.xml
new file mode 100644
index 0000000..1591d3f
--- /dev/null
+++ b/docs/topics/impala_runtime_filter_max_size.xml
@@ -0,0 +1,51 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="runtime_filter_max_size" rev="2.6.0 IMPALA-3480 CDH-41184">
+
+  <title>RUNTIME_FILTER_MAX_SIZE Query Option (<keyword keyref="impala26"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>RUNTIME_FILTER_MAX_SIZE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.6.0 IMPALA-3480">
+      <indexterm audience="Cloudera">RUNTIME_FILTER_MAX_SIZE query option</indexterm>
+      The <codeph>RUNTIME_FILTER_MAX_SIZE</codeph> query option
+      adjusts the settings for the runtime filtering feature.
+      This option defines the maximum size for a filter,
+      no matter what the estimates produced by the planner are.
+      This value also overrides any lower number specified for the
+      <codeph>RUNTIME_BLOOM_FILTER_SIZE</codeph> query option.
+      Filter sizes are rounded up to the nearest power of two.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_integer"/>
+
+    <p>
+      <b>Default:</b> 0 (meaning use the value from the corresponding <cmdname>impalad</cmdname> startup option)
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/added_in_260"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/runtime_filtering_option_caveat"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_runtime_filtering.xml"/>,
+      <xref href="impala_runtime_filter_mode.xml#runtime_filter_mode"/>,
+      <xref href="impala_runtime_filter_min_size.xml#runtime_filter_min_size"/>,
+      <xref href="impala_runtime_bloom_filter_size.xml#runtime_bloom_filter_size"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_runtime_filter_min_size.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_runtime_filter_min_size.xml b/docs/topics/impala_runtime_filter_min_size.xml
new file mode 100644
index 0000000..75944e3
--- /dev/null
+++ b/docs/topics/impala_runtime_filter_min_size.xml
@@ -0,0 +1,51 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="runtime_filter_min_size" rev="2.6.0 IMPALA-3480 CDH-41184">
+
+  <title>RUNTIME_FILTER_MIN_SIZE Query Option (<keyword keyref="impala26"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>RUNTIME_FILTER_MIN_SIZE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.6.0 IMPALA-3480">
+      <indexterm audience="Cloudera">RUNTIME_FILTER_MIN_SIZE query option</indexterm>
+      The <codeph>RUNTIME_FILTER_MIN_SIZE</codeph> query option
+      adjusts the settings for the runtime filtering feature.
+      This option defines the minimum size for a filter,
+      no matter what the estimates produced by the planner are.
+      This value also overrides any lower number specified for the
+      <codeph>RUNTIME_BLOOM_FILTER_SIZE</codeph> query option.
+      Filter sizes are rounded up to the nearest power of two.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_integer"/>
+
+    <p>
+      <b>Default:</b> 0 (meaning use the value from the corresponding <cmdname>impalad</cmdname> startup option)
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/added_in_260"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/runtime_filtering_option_caveat"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_runtime_filtering.xml"/>,
+      <xref href="impala_runtime_filter_mode.xml#runtime_filter_mode"/>,
+      <xref href="impala_runtime_filter_max_size.xml#runtime_filter_max_size"/>,
+      <xref href="impala_runtime_bloom_filter_size.xml#runtime_bloom_filter_size"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_runtime_filter_mode.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_runtime_filter_mode.xml b/docs/topics/impala_runtime_filter_mode.xml
new file mode 100644
index 0000000..fa75bdf
--- /dev/null
+++ b/docs/topics/impala_runtime_filter_mode.xml
@@ -0,0 +1,77 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="runtime_filter_mode" rev="2.5.0">
+
+  <title>RUNTIME_FILTER_MODE Query Option (<keyword keyref="impala25"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>RUNTIME_FILTER_MODE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.5.0">
+      <indexterm audience="Cloudera">RUNTIME_FILTER_MODE query option</indexterm>
+    </p>
+
+    <p>
+      The <codeph>RUNTIME_FILTER_MODE</codeph> query option
+      adjusts the settings for the runtime filtering feature.
+      It turns this feature on and off, and controls how
+      extensively the filters are transmitted between hosts.
+    </p>
+
+    <p>
+      <b>Type:</b> numeric (0, 1, 2)
+      or corresponding mnemonic strings (<codeph>OFF</codeph>, <codeph>LOCAL</codeph>, <codeph>GLOBAL</codeph>).
+    </p>
+
+    <p rev="2.6.0 CDH-41184">
+      <b>Default:</b> 2 (equivalent to <codeph>GLOBAL</codeph>); formerly was 1 / <codeph>LOCAL</codeph>, in <keyword keyref="impala25_full"/>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/added_in_250"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p rev="2.6.0 CDH-41184">
+      In <keyword keyref="impala26_full"/> and higher, the default is <codeph>GLOBAL</codeph>.
+      This setting is recommended for a wide variety of workloads, to provide best
+      performance with <q>out of the box</q> settings.
+    </p>
+
+    <p rev="2.6.0 CDH-41184">
+      The lowest setting of <codeph>LOCAL</codeph> does a similar level of optimization
+      (such as partition pruning) as in earlier Impala releases.
+      This setting was the default in <keyword keyref="impala25_full"/>,
+      to allow for a period of post-upgrade testing for existing workloads.
+      This setting is suitable for workloads with non-performance-critical queries,
+      or if the coordinator node is under heavy CPU or memory pressure.
+    </p>
+
+    <p>
+      You might change the setting to <codeph>OFF</codeph> if your workload contains
+      many queries involving partitioned tables or joins that do not experience a performance
+      increase from the runtime filters feature. If the overhead of producing the runtime filters
+      outweighs the performance benefit for queries, you can turn the feature off entirely.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_partitioning.xml#partitioning"/> for details about runtime filtering.
+      <xref href="impala_disable_row_runtime_filtering.xml#disable_row_runtime_filtering"/>,
+      <xref href="impala_runtime_bloom_filter_size.xml#runtime_bloom_filter_size"/>,
+      <xref href="impala_runtime_filter_wait_time_ms.xml#runtime_filter_wait_time_ms"/>,
+      and
+      <xref href="impala_max_num_runtime_filters.xml#max_num_runtime_filters"/>
+      for tuning options for runtime filtering.
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_runtime_filter_wait_time_ms.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_runtime_filter_wait_time_ms.xml b/docs/topics/impala_runtime_filter_wait_time_ms.xml
new file mode 100644
index 0000000..fb41885
--- /dev/null
+++ b/docs/topics/impala_runtime_filter_wait_time_ms.xml
@@ -0,0 +1,47 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="runtime_filter_wait_time_ms" rev="2.5.0">
+
+  <title>RUNTIME_FILTER_WAIT_TIME_MS Query Option (<keyword keyref="impala25"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>RUNTIME_FILTER_WAIT_TIME_MS</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.5.0">
+      <indexterm audience="Cloudera">RUNTIME_FILTER_WAIT_TIME_MS query option</indexterm>
+      The <codeph>RUNTIME_FILTER_WAIT_TIME_MS</codeph> query option
+      adjusts the settings for the runtime filtering feature.
+      It specifies a time in milliseconds that each scan node waits for
+      runtime filters to be produced by other plan fragments.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_integer"/>
+
+    <p>
+      <b>Default:</b> 0 (meaning use the value from the corresponding <cmdname>impalad</cmdname> startup option)
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/added_in_250"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/runtime_filtering_option_caveat"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_runtime_filtering.xml"/>,
+      <xref href="impala_runtime_filter_mode.xml#runtime_filter_mode"/>
+      <!-- , <xref href="impala_partitioning.xml#dynamic_partition_pruning"/> -->
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_runtime_filtering.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_runtime_filtering.xml b/docs/topics/impala_runtime_filtering.xml
new file mode 100644
index 0000000..b5c1fcb
--- /dev/null
+++ b/docs/topics/impala_runtime_filtering.xml
@@ -0,0 +1,506 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="runtime_filtering" rev="2.5.0">
+
+  <title id="runtime_filters">Runtime Filtering for Impala Queries (<keyword keyref="impala25"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>Runtime Filtering</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.5.0">
+      <indexterm audience="Cloudera">runtime filtering</indexterm>
+      <term>Runtime filtering</term> is a wide-ranging optimization feature available in
+      <keyword keyref="impala25_full"/> and higher. When only a fraction of the data in a table is
+      needed for a query against a partitioned table or to evaluate a join condition,
+      Impala determines the appropriate conditions while the query is running, and
+      broadcasts that information to all the <cmdname>impalad</cmdname> nodes that are reading the table
+      so that they can avoid unnecessary I/O to read partition data, and avoid
+      unnecessary network transmission by sending only the subset of rows that match the join keys
+      across the network.
+    </p>
+
+    <p>
+      This feature is primarily used to optimize queries against large partitioned tables
+      (under the name <term>dynamic partition pruning</term>) and joins of large tables.
+      The information in this section includes concepts, internals, and troubleshooting
+      information for the entire runtime filtering feature.
+      For specific tuning steps for partitioned tables,
+      <!-- and join queries, -->
+      see
+      <xref href="impala_partitioning.xml#dynamic_partition_pruning"/>.
+      <!-- and <xref href="impala_joins.xml#joins"/>. -->
+    </p>
+
+    <note type="important" rev="2.6.0 CDH-41184">
+      <p rev="2.6.0 CDH-41184">
+        When this feature made its debut in CDH 5.7 / Impala 2.5,
+        the default setting was <codeph>RUNTIME_FILTER_MODE=LOCAL</codeph>.
+        Now the default is <codeph>RUNTIME_FILTER_MODE=GLOBAL</codeph> in <keyword keyref="impala26_full"/> and higher,
+        which enables more wide-ranging and ambitious query optimization without requiring you to
+        explicitly set any query options.
+      </p>
+    </note>
+
+    <p outputclass="toc inpage"/>
+
+  </conbody>
+
+  <concept id="runtime_filtering_concepts">
+    <title>Background Information for Runtime Filtering</title>
+    <conbody>
+      <p>
+        To understand how runtime filtering works at a detailed level, you must
+        be familiar with some terminology from the field of distributed database technology:
+      </p>
+      <ul>
+        <li>
+          <p>
+            What a <term>plan fragment</term> is.
+            Impala decomposes each query into smaller units of work that are distributed across the cluster.
+            Wherever possible, a data block is read, filtered, and aggregated by plan fragments executing
+            on the same host. For some operations, such as joins and combining intermediate results into
+            a final result set, data is transmitted across the network from one DataNode to another.
+          </p>
+        </li>
+        <li>
+          <p>
+            What <codeph>SCAN</codeph> and <codeph>HASH JOIN</codeph> plan nodes are, and their role in computing query results:
+          </p>
+          <p>
+            In the Impala query plan, a <term>scan node</term> performs the I/O to read from the underlying data files.
+            Although this is an expensive operation from the traditional database perspective, Hadoop clusters and Impala are
+            optimized to do this kind of I/O in a highly parallel fashion. The major potential cost savings come from using
+            the columnar Parquet format (where Impala can avoid reading data for unneeded columns) and partitioned tables
+            (where Impala can avoid reading data for unneeded partitions).
+          </p>
+          <p>
+            Most Impala joins use the
+            <xref href="https://en.wikipedia.org/wiki/Hash_join" scope="external" format="html"><term>hash join</term></xref>
+            mechanism. (It is only fairly recently that Impala
+            started using the nested-loop join technique, for certain kinds of non-equijoin queries.)
+            In a hash join, when evaluating join conditions from two tables, Impala constructs a hash table in memory with all
+            the different column values from the table on one side of the join.
+            Then, for each row from the table on the other side of the join, Impala tests whether the relevant column values
+            are in this hash table or not.
+          </p>
+          <p>
+            A <term>hash join node</term> constructs such an in-memory hash table, then performs the comparisons to
+            identify which rows match the relevant join conditions
+            and should be included in the result set (or at least sent on to the subsequent intermediate stage of
+            query processing). Because some of the input for a hash join might be transmitted across the network from another host,
+            it is especially important from a performance perspective to prune out ahead of time any data that is known to be
+            irrelevant.
+          </p>
+          <p>
+            The more distinct values are in the columns used as join keys, the larger the in-memory hash table and
+            thus the more memory required to process the query.
+          </p>
+        </li>
+        <li>
+          <p>
+            The difference between a <term>broadcast join</term> and a <term>shuffle join</term>.
+            (The Hadoop notion of a shuffle join is sometimes referred to in Impala as a <term>partitioned join</term>.)
+            In a broadcast join, the table from one side of the join (typically the smaller table)
+            is sent in its entirety to all the hosts involved in the query. Then each host can compare its
+            portion of the data from the other (larger) table against the full set of possible join keys.
+            In a shuffle join, there is no obvious <q>smaller</q> table, and so the contents of both tables
+            are divided up, and corresponding portions of the data are transmitted to each host involved in the query.
+            See <xref href="impala_hints.xml#hints"/> for information about how these different kinds of
+            joins are processed.
+          </p>
+        </li>
+        <li>
+          <p>
+            The notion of the build phase and probe phase when Impala processes a join query.
+            The <term>build phase</term> is where the rows containing the join key columns, typically for the smaller table,
+            are transmitted across the network and built into an in-memory hash table data structure on one or
+            more destination nodes.
+            The <term>probe phase</term> is where data is read locally (typically from the larger table) and the join key columns
+            are compared to the values in the in-memory hash table.
+            The corresponding input sources (tables, subqueries, and so on) for these
+            phases are referred to as the <term>build side</term> and the <term>probe side</term>.
+          </p>
+        </li>
+        <li>
+          <p>
+            How to set Impala query options: interactively within an <cmdname>impala-shell</cmdname> session through
+            the <codeph>SET</codeph> command, for a JDBC or ODBC application through the <codeph>SET</codeph> statement, or
+            globally for all <cmdname>impalad</cmdname> daemons through the <codeph>default_query_options</codeph> configuration
+            setting.
+          </p>
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="runtime_filtering_internals">
+    <title>Runtime Filtering Internals</title>
+    <conbody>
+      <p>
+        The <term>filter</term> that is transmitted between plan fragments is essentially a list
+        of values for join key columns. When this list is values is transmitted in time to a scan node,
+        Impala can filter out non-matching values immediately after reading them, rather than transmitting
+        the raw data to another host to compare against the in-memory hash table on that host.
+        This data structure is implemented as a <term>Bloom filter</term>, which uses a probability-based
+        algorithm to determine all possible matching values. (The probability-based aspects means that the
+        filter might include some non-matching values, but if so, that does not cause any inaccuracy
+        in the final results.)
+      </p>
+      <p>
+        There are different kinds of filters to match the different kinds of joins (partitioned and broadcast).
+        A broadcast filter is a complete list of relevant values that can be immediately evaluated by a scan node.
+        A partitioned filter is a partial list of relevant values (based on the data processed by one host in the
+        cluster); all the partitioned filters must be combined into one (by the coordinator node) before the
+        scan nodes can use the results to accurately filter the data as it is read from storage.
+      </p>
+      <p>
+        Broadcast filters are also classified as local or global. With a local broadcast filter, the information
+        in the filter is used by a subsequent query fragment that is running on the same host that produced the filter.
+        A non-local broadcast filter must be transmitted across the network to a query fragment that is running on a
+        different host. Impala designates 3 hosts to each produce non-local broadcast filters, to guard against the
+        possibility of a single slow host taking too long. Depending on the setting of the <codeph>RUNTIME_FILTER_MODE</codeph> query option
+        (<codeph>LOCAL</codeph> or <codeph>GLOBAL</codeph>), Impala either uses a conservative optimization
+        strategy where filters are only consumed on the same host that produced them, or a more aggressive strategy
+        where filters are eligible to be transmitted across the network.
+      </p>
+
+      <note rev="2.6.0 CDH-41184 IMPALA-3333">
+        In <keyword keyref="impala26_full"/> and higher, the default for runtime filtering is the <codeph>GLOBAL</codeph> setting.
+      </note>
+
+    </conbody>
+  </concept>
+
+  <concept id="runtime_filtering_file_formats">
+    <title>File Format Considerations for Runtime Filtering</title>
+    <conbody>
+      <p>
+        Parquet tables get the most benefit from
+        the runtime filtering optimizations. Runtime filtering can speed up
+        join queries against partitioned or unpartitioned Parquet tables,
+        and single-table queries against partitioned Parquet tables.
+        See <xref href="impala_parquet.xml#parquet"/> for information about
+        using Parquet tables with Impala.
+      </p>
+      <p>
+        For other file formats (text, Avro, RCFile, and SequenceFile),
+        runtime filtering speeds up queries against partitioned tables only.
+        Because partitioned tables can use a mixture of formats, Impala produces
+        the filters in all cases, even if they are not ultimately used to
+        optimize the query.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="runtime_filtering_timing">
+    <title>Wait Intervals for Runtime Filters</title>
+    <conbody>
+      <p>
+        Because it takes time to produce runtime filters, especially for
+        partitioned filters that must be combined by the coordinator node,
+        there is a time interval above which it is more efficient for
+        the scan nodes to go ahead and construct their intermediate result sets,
+        even if that intermediate data is larger than optimal. If it only takes
+        a few seconds to produce the filters, it is worth the extra time if pruning
+        the unnecessary data can save minutes in the overall query time.
+        You can specify the maximum wait time in milliseconds using the
+        <codeph>RUNTIME_FILTER_WAIT_TIME_MS</codeph> query option.
+      </p>
+      <p>
+        By default, each scan node waits for up to 1 second (1000 milliseconds)
+        for filters to arrive. If all filters have not arrived within the
+        specified interval, the scan node proceeds, using whatever filters
+        did arrive to help avoid reading unnecessary data. If a filter arrives
+        after the scan node begins reading data, the scan node applies that
+        filter to the data that is read after the filter arrives, but not to
+        the data that was already read.
+      </p>
+      <p>
+        If the cluster is relatively busy and your workload contains many
+        resource-intensive or long-running queries, consider increasing the wait time
+        so that complicated queries do not miss opportunities for optimization.
+        If the cluster is lightly loaded and your workload contains many small queries
+        taking only a few seconds, consider decreasing the wait time to avoid the
+        1 second delay for each query.
+      </p>
+    </conbody>
+  </concept>
+
+
+  <concept id="runtime_filtering_query_options">
+    <title>Query Options for Runtime Filtering</title>
+    <conbody>
+      <p>
+        See the following sections for information about the query options that control runtime filtering:
+      </p>
+      <ul>
+        <li>
+          <p>
+            The first query option adjusts the <q>sensitivity</q> of this feature.
+            <ph rev="2.6.0 CDH-41184 IMPALA-3333">By default, it is set to the highest level (<codeph>GLOBAL</codeph>).
+            (This default applies to <keyword keyref="impala26_full"/> and higher.
+            In previous releases, the default was <codeph>LOCAL</codeph>.)</ph>
+          </p>
+          <ul>
+            <li>
+              <p>
+                <xref href="impala_runtime_filter_mode.xml#runtime_filter_mode"/>
+              </p>
+            </li>
+          </ul>
+        </li>
+        <li>
+          <p>
+            The other query options are tuning knobs that you typically only adjust after doing
+            performance testing, and that you might want to change only for the duration of a single
+            expensive query:
+          </p>
+          <ul>
+            <li>
+              <p>
+                <xref href="impala_max_num_runtime_filters.xml#max_num_runtime_filters"/>
+              </p>
+            </li>
+            <li>
+              <p>
+                <xref href="impala_disable_row_runtime_filtering.xml#disable_row_runtime_filtering"/>
+              </p>
+            </li>
+            <li>
+              <p rev="2.6.0 IMPALA-3480">
+                <xref href="impala_runtime_filter_max_size.xml#runtime_filter_max_size"/> 
+              </p>
+            </li>
+            <li>
+              <p rev="2.6.0 IMPALA-3480">
+                <xref href="impala_runtime_filter_min_size.xml#runtime_filter_min_size"/> 
+              </p>
+            </li>
+            <li>
+              <p rev="2.6.0 IMPALA-3007">
+                <xref href="impala_runtime_bloom_filter_size.xml#runtime_bloom_filter_size"/>;
+                in <keyword keyref="impala26_full"/> and higher, this setting acts as a fallback when
+                statistics are not available, rather than as a directive.
+              </p>
+            </li>
+          </ul>
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="runtime_filtering_explain_plan">
+    <title>Runtime Filtering and Query Plans</title>
+    <conbody>
+      <p>
+        In the same way the query plan displayed by the
+        <codeph>EXPLAIN</codeph> statement includes information
+        about predicates used by each plan fragment, it also
+        includes annotations showing whether a plan fragment
+        produces or consumes a runtime filter.
+        A plan fragment that produces a filter includes an
+        annotation such as
+        <codeph>runtime filters: <varname>filter_id</varname> &lt;- <varname>table</varname>.<varname>column</varname></codeph>,
+        while a plan fragment that consumes a filter includes an annotation such as
+        <codeph>runtime filters: <varname>filter_id</varname> -&gt; <varname>table</varname>.<varname>column</varname></codeph>.
+      </p>
+
+      <p>
+        The following example shows a query that uses a single runtime filter (labelled <codeph>RF00</codeph>)
+        to prune the partitions that are scanned in one stage of the query, based on evaluating the
+        result set of a subquery:
+      </p>
+
+<codeblock conref="../shared/impala_common.xml#common/simple_dpp_example"/>
+
+      <p>
+        The query profile (displayed by the <codeph>PROFILE</codeph> command in <cmdname>impala-shell</cmdname>)
+        contains both the <codeph>EXPLAIN</codeph> plan and more detailed information about the internal
+        workings of the query. The profile output includes a section labelled the <q>filter routing table</q>,
+        with information about each filter based on its ID.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="runtime_filtering_queries">
+    <title>Examples of Queries that Benefit from Runtime Filtering</title>
+    <conbody>
+
+      <p>
+        In this example, Impala would normally do extra work to interpret the columns
+        <codeph>C1</codeph>, <codeph>C2</codeph>, <codeph>C3</codeph>, and <codeph>ID</codeph>
+        for each row in <codeph>HUGE_T1</codeph>, before checking the <codeph>ID</codeph>
+        value against the in-memory hash table constructed from all the <codeph>TINY_T2.ID</codeph>
+        values. By producing a filter containing all the <codeph>TINY_T2.ID</codeph> values
+        even before the query starts scanning the <codeph>HUGE_T1</codeph> table, Impala
+        can skip the unnecessary work to parse the column info as soon as it determines
+        that an <codeph>ID</codeph> value does not match any of the values from the other table.
+      </p>
+
+      <p>
+        The example shows <codeph>COMPUTE STATS</codeph> statements for both the tables (even
+        though that is a one-time operation after loading data into those tables) because
+        Impala relies on up-to-date statistics to
+        determine which one has more distinct <codeph>ID</codeph> values than the other.
+        That information lets Impala make effective decisions about which table to use to
+        construct the in-memory hash table, and which table to read from disk and
+        compare against the entries in the hash table.
+      </p>
+
+<codeblock rev="2.6.0 CDH-41184">
+COMPUTE STATS huge_t1;
+COMPUTE STATS tiny_t2;
+SELECT c1, c2, c3 FROM huge_t1 JOIN tiny_t2 WHERE huge_t1.id = tiny_t2.id;
+</codeblock>
+
+<!-- The greater-than comparison prevents runtime filtering from applying. Comment out for now;
+     put back if the example can be reworked in a way that does produce some filters.
+      <p>
+        In this example, <codeph>T1</codeph> is a table partitioned by year. The subquery
+        on <codeph>T2</codeph> produces a single value with the <codeph>MIN(year)</codeph> result,
+        and transmits that value as a filter to the plan fragments that are reading from <codeph>T1</codeph>.
+        Any non-matching partitions in <codeph>T1</codeph> are skipped.
+      </p>
+
+<codeblock>
+select c1 from t1 where year > (select min(year) from t2);
+</codeblock>
+-->
+
+      <p>
+        In this example, <codeph>T1</codeph> is a table partitioned by year. The subquery
+        on <codeph>T2</codeph> produces multiple values, and transmits those values as a filter to the plan
+        fragments that are reading from <codeph>T1</codeph>. Any non-matching partitions in <codeph>T1</codeph>
+        are skipped.
+      </p>
+
+<codeblock rev="2.6.0 CDH-41184">
+select c1 from t1 where year in (select distinct year from t2);
+</codeblock>
+
+      <p>
+        Now the <codeph>WHERE</codeph> clause contains an additional test that does not apply to
+        the partition key column.
+        A filter on a column that is not a partition key is called a per-row filter.
+        Because per-row filters only apply for Parquet, <codeph>T1</codeph> must be a Parquet table.
+      </p>
+
+      <p>
+        The subqueries result in two filters being transmitted to
+        the scan nodes that read from <codeph>T1</codeph>. The filter on <codeph>YEAR</codeph> helps the query eliminate
+        entire partitions based on non-matching years. The filter on <codeph>C2</codeph> lets Impala discard
+        rows with non-matching <codeph>C2</codeph> values immediately after reading them. Without runtime filtering,
+        Impala would have to keep the non-matching values in memory, assemble <codeph>C1</codeph>, <codeph>C2</codeph>,
+        and <codeph>C3</codeph> into rows in the intermediate result set, and transmit all the intermediate rows
+        back to the coordinator node, where they would be eliminated only at the very end of the query.
+      </p>
+
+<codeblock rev="2.6.0 CDH-41184">
+select c1, c2, c3 from t1
+  where year in (select distinct year from t2)
+    and c2 in (select other_column from t3);
+</codeblock>
+
+      <p>
+        This example involves a broadcast join.
+        The fact that the <codeph>ON</codeph> clause would
+        return a small number of matching rows (because there
+        are not very many rows in <codeph>TINY_T2</codeph>)
+        means that the corresponding filter is very selective.
+        Therefore, runtime filtering will probably be effective
+        in optimizing this query.
+      </p>
+
+<codeblock rev="2.6.0 CDH-41184">
+select c1 from huge_t1 join [broadcast] tiny_t2
+  on huge_t1.id = tiny_t2.id
+  where huge_t1.year in (select distinct year from tiny_t2)
+    and c2 in (select other_column from t3);
+</codeblock>
+
+      <p>
+        This example involves a shuffle or partitioned join.
+        Assume that most rows in <codeph>HUGE_T1</codeph>
+        have a corresponding row in <codeph>HUGE_T2</codeph>.
+        The fact that the <codeph>ON</codeph> clause could
+        return a large number of matching rows means that
+        the corresponding filter would not be very selective.
+        Therefore, runtime filtering might be less effective
+        in optimizing this query.
+      </p>
+
+<codeblock rev="2.6.0 CDH-41184">
+select c1 from huge_t1 join [shuffle] huge_t2
+  on huge_t1.id = huge_t2.id
+  where huge_t1.year in (select distinct year from huge_t2)
+    and c2 in (select other_column from t3);
+</codeblock>
+
+    </conbody>
+  </concept>
+
+  <concept id="runtime_filtering_tuning">
+    <title>Tuning and Troubleshooting Queries that Use Runtime Filtering</title>
+    <conbody>
+      <p>
+        These tuning and troubleshooting procedures apply to queries that are
+        resource-intensive enough, long-running enough, and frequent enough
+        that you can devote special attention to optimizing them individually.
+      </p>
+
+      <p>
+        Use the <codeph>EXPLAIN</codeph> statement and examine the <codeph>runtime filters:</codeph>
+        lines to determine whether runtime filters are being applied to the <codeph>WHERE</codeph> predicates
+        and join clauses that you expect. For example, runtime filtering does not apply to queries that use
+        the nested loop join mechanism due to non-equijoin operators.
+      </p>
+
+      <p>
+        Make sure statistics are up-to-date for all tables involved in the queries.
+        Use the <codeph>COMPUTE STATS</codeph> statement after loading data into non-partitioned tables,
+        and <codeph>COMPUTE INCREMENTAL STATS</codeph> after adding new partitions to partitioned tables.
+      </p>
+
+      <p>
+        If join queries involving large tables use unique columns as the join keys,
+        for example joining a primary key column with a foreign key column, the overhead of
+        producing and transmitting the filter might outweigh the performance benefit because
+        not much data could be pruned during the early stages of the query.
+        For such queries, consider setting the query option <codeph>RUNTIME_FILTER_MODE=OFF</codeph>.
+      </p>
+
+    </conbody>
+  </concept>
+
+  <concept id="runtime_filtering_limits">
+    <title>Limitations and Restrictions for Runtime Filtering</title>
+    <conbody>
+      <p>
+        The runtime filtering feature is most effective for the Parquet file formats.
+        For other file formats, filtering only applies for partitioned tables.
+        See <xref href="impala_runtime_filtering.xml#runtime_filtering_file_formats"/>.
+      </p>
+
+      <!-- To do: check if this restriction is lifted in 5.8 / 2.6. -->
+      <p rev="IMPALA-3054">
+        When the spill-to-disk mechanism is activated on a particular host during a query,
+        that host does not produce any filters while processing that query.
+        This limitation does not affect the correctness of results; it only reduces the
+        amount of optimization that can be applied to the query.
+      </p>
+
+    </conbody>
+  </concept>
+
+
+</concept>


[18/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_new_features.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_new_features.xml b/docs/topics/impala_new_features.xml
new file mode 100644
index 0000000..4da811f
--- /dev/null
+++ b/docs/topics/impala_new_features.xml
@@ -0,0 +1,4015 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="ver" id="new_features">
+
+  <title><ph audience="standalone">New Features in Apache Impala (incubating)</ph><ph audience="integrated">What's New in Apache Impala (incubating)</ph></title>
+
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Release Notes"/>
+      <data name="Category" value="New Features"/>
+      <data name="Category" value="What's New"/>
+      <data name="Category" value="Getting Started"/>
+      <data name="Category" value="Upgrading"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      This release of Impala contains the following changes and enhancements from previous releases.
+    </p>
+
+    <p outputclass="toc inpage"/>
+
+  </conbody>
+
+<!-- All 2.7.x new features go under here -->
+
+  <concept rev="2.7.0" id="new_features_270">
+
+    <title>New Features in Impala 2.7.x / CDH 5.9.x</title>
+
+    <conbody>
+
+      <ul id="feature_list">
+        <li>
+          <p>
+            Performance improvements:
+          </p>
+          <ul>
+            <li>
+              <p rev="IMPALA-3206 CDH-43744">
+                [<xref href="https://issues.cloudera.org/browse/IMPALA-3206" scope="external" format="html">IMPALA-3206</xref>]
+                Speedup for queries against <codeph>DECIMAL</codeph> columns in Avro tables.
+                The code that parses <codeph>DECIMAL</codeph> values from Avro now uses
+                native code generation.
+              </p>
+            </li>
+            <li>
+              <p rev="IMPALA-3674">
+                [<xref href="https://issues.cloudera.org/browse/IMPALA-3674" scope="external" format="html">IMPALA-3674</xref>]
+                Improved efficiency in LLVM code generation can reduce codegen time, especially
+                for short queries.
+              </p>
+            </li>
+            <!-- Not actually a new feature, it's more a tip about when to expect remote reads and how to minimize them. To go somewhere in the performance / best practices / Parquet info.
+            <li>
+              <p rev="IMPALA-3885 CDH-43793">
+                [<xref href="https://issues.cloudera.org/browse/IMPALA-3885" scope="external" format="html">IMPALA-3885</xref>]
+                Parquet files with multiple blocks can now be processed
+                without remote reads.
+              </p>
+            </li>
+            -->
+            <li>
+              <p rev="IMPALA-2979 CDH-43739"> [<xref
+                  href="https://issues.cloudera.org/browse/IMPALA-2979" scope="external"
+                  format="html">IMPALA-2979</xref>] Improvements to scheduling on worker nodes,
+                enabled by the <codeph>REPLICA_PREFERENCE</codeph> query option.
+                See <xref
+                  href="impala_replica_preference.xml#replica_preference"/> for details.
+              </p>
+            </li>
+          </ul>
+        </li>
+        <li audience="Cloudera">
+          <p rev="IMPALA-3210 CDH-43736"><!-- Patch didn't make it into in <keyword keyref="impala27_full"/> -->
+            [<xref href="https://issues.cloudera.org/browse/IMPALA-3210" scope="external" format="html">IMPALA-3210</xref>]
+            The analytic functions <codeph>FIRST_VALUE()</codeph> and <codeph>LAST_VALUE()</codeph>
+            accept a new clause, <codeph>IGNORE NULLS</codeph>.
+            See <xref href="impala_analytic_functions.xml#first_value"/>
+            and <xref href="impala_analytic_functions.xml#last_value"/>
+            for details.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-1683 CDH-43732">
+            [<xref href="https://issues.cloudera.org/browse/IMPALA-1683" scope="external" format="html">IMPALA-1683</xref>]
+            The <codeph>REFRESH</codeph> statement can be applied to a single partition,
+            rather than the entire table. See <xref href="impala_refresh.xml#refresh"/>
+            and <xref href="impala_partitioning.xml#partition_refresh"/> for details.
+          </p>
+        </li>
+        <li>
+          <p>
+            Improvements to the Impala web user interface:
+          </p>
+          <ul>
+            <li>
+              <p rev="IMPALA-2767 CDH-43748">
+                [<xref href="https://issues.cloudera.org/browse/IMPALA-2767" scope="external" format="html">IMPALA-2767</xref>]
+                You can now force a session to expire by clicking a link in the web UI,
+                on the <uicontrol>/sessions</uicontrol> tab.
+              </p>
+            </li>
+            <li>
+              <p rev="IMPALA-3715 CDH-43743">
+                [<xref href="https://issues.cloudera.org/browse/IMPALA-3715" scope="external" format="html">IMPALA-3715</xref>]
+                The <uicontrol>/memz</uicontrol> tab includes more information about
+                Impala memory usage.
+              </p>
+            </li>
+            <li>
+              <p rev="IMPALA-3716 CDH-43741">
+                [<xref href="https://issues.cloudera.org/browse/IMPALA-3716" scope="external" format="html">IMPALA-3716</xref>]
+                The <uicontrol>Details</uicontrol> page for a query now includes
+                a <uicontrol>Memory</uicontrol> tab.
+              </p>
+            </li>
+          </ul>
+        </li>
+        <li>
+          <p rev="IMPALA-3499 CDH-43740">
+            [<xref href="https://issues.cloudera.org/browse/IMPALA-3499" scope="external" format="html">IMPALA-3499</xref>]
+            Scalability improvements to the catalog server. Impala handles internal communication
+            more efficiently for tables with large numbers of columns and partitions, where the
+            size of the metadata exceeds 2 GiB.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-3677 CDH-43745">
+            [<xref href="https://issues.cloudera.org/browse/IMPALA-3677" scope="external" format="html">IMPALA-3677</xref>]
+            You can send a <codeph>SIGUSR1</codeph> signal to any Impala-related daemon to write a
+            Breakpad minidump. For advanced troubleshooting, you can now produce a minidump
+            without triggering a crash. See <xref href="impala_breakpad.xml#breakpad"/> for
+            details about the Breakpad minidump feature.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-3687 CDH-43731">
+            [<xref href="https://issues.cloudera.org/browse/IMPALA-3687" scope="external" format="html">IMPALA-3687</xref>]
+            The schema reconciliation rules for Avro tables have changed slightly
+            for <codeph>CHAR</codeph> and <codeph>VARCHAR</codeph> columns. Now, if
+            the definition of such a column is changed in the Avro schema file,
+            the column retains its <codeph>CHAR</codeph> or <codeph>VARCHAR</codeph>
+            type as specified in the SQL definition, but the column name and comment
+            from the Avro schema file take precedence.
+            See <xref href="impala_avro.xml#avro_create_table"/> for details about
+            column definitions in Avro tables.
+          </p>
+        </li>
+        <li audience="Cloudera"><!-- Patch didn't make it into in <keyword keyref="impala27_full"/> -->
+          <p rev="IMPALA-1654 CDH-43747">
+            [<xref href="https://issues.cloudera.org/browse/IMPALA-1654" scope="external" format="html">IMPALA-1654</xref>]
+            Several kinds of DDL operations
+            can now work on a range of partitions. The partitions can be specified
+            using operators such as <codeph>&lt;</codeph>, <codeph>&gt;=</codeph>, and
+            <codeph>!=</codeph> rather than just an equality predicate applying to a single
+            partition.
+            This new feature extends the syntax of
+            several clauses
+            of the <codeph>ALTER TABLE</codeph> statement
+            (<codeph>DROP PARTITION</codeph>, <codeph>SET [UN]CACHED</codeph>,
+            <codeph>SET FILEFORMAT | SERDEPROPERTIES | TBLPROPERTIES</codeph>),
+            the <codeph>SHOW FILES</codeph> statement, and the
+            <codeph>COMPUTE INCREMENTAL STATS</codeph> statement.
+            It does not apply to statements that are defined to only apply to a single
+            partition, such as <codeph>LOAD DATA</codeph>, <codeph>ALTER TABLE ... ADD PARTITION</codeph>,
+            <codeph>SET LOCATION</codeph>, and <codeph>INSERT</codeph> with a static
+            partitioning clause.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-3575 CDH-43742"> [<xref
+              href="https://issues.cloudera.org/browse/IMPALA-3575"
+              scope="external" format="html">IMPALA-3575</xref>] Some network
+            operations now have additional timeout and retry settings. The extra
+            configuration helps avoid failed queries for transient network
+            problems, to avoid hangs when a sender or receiver fails in the
+            middle of a network transmission, and to make cancellation requests
+            more reliable despite network issues. </p>
+        </li>
+      </ul>
+
+    </conbody>
+  </concept>
+<!-- All 2.6.x new features go under here -->
+
+  <concept rev="2.6.0" id="new_features_260">
+
+    <title>New Features in Impala 2.6.x / CDH 5.8.x</title>
+
+    <conbody>
+
+      <!-- <note conref="../shared/impala_common.xml#common/only_cdh5_260" /> -->
+
+      <ul>
+        <li>
+          <p>
+            Improvements to Impala support for the Amazon S3 filesystem:
+          </p>
+          <ul>
+            <li>
+              <p rev="IMPALA-1878 CDH-33310">
+                Impala can now write to S3 tables through the <codeph>INSERT</codeph>
+                or <codeph>LOAD DATA</codeph> statements.
+                See <xref href="impala_s3.xml#s3"/> for general information about
+                using Impala with S3.
+              </p>
+            </li>
+            <li>
+              <p rev="IMPALA-3452 CDH-39913">
+                A new query option, <codeph>S3_SKIP_INSERT_STAGING</codeph>, lets you
+                trade off between fast <codeph>INSERT</codeph> performance and
+                slower <codeph>INSERT</codeph>s that are more consistent if a
+                problem occurs during the statement. The new behavior is enabled by default.
+                See <xref href="impala_s3_skip_insert_staging.xml#s3_skip_insert_staging"/> for details
+                about this option.
+              </p>
+            </li>
+          </ul>
+        </li>
+        <li>
+          <p rev="CDH-41184">
+            Performance improvements for the runtime filtering feature:
+          </p>
+          <ul>
+            <li>
+              <p rev="CDH-41184 IMPALA-3333">
+                The default for the <codeph>RUNTIME_FILTER_MODE</codeph> 
+                query option is changed to <codeph>GLOBAL</codeph> (the highest setting).
+                See <xref href="impala_runtime_filter_mode.xml#runtime_filter_mode"/> for
+                details about this option.
+              </p>
+            </li>
+            <li rev="CDH-41184 IMPALA-3007">
+              <p>
+                The <codeph>RUNTIME_BLOOM_FILTER_SIZE</codeph> setting is now only used
+                as a fallback if statistics are not available; otherwise, Impala
+                uses the statistics to estimate the appropriate size to use for each filter.
+                See <xref href="impala_runtime_bloom_filter_size.xml#runtime_bloom_filter_size"/> for
+                details about this option.
+              </p>
+            </li>
+            <li rev="CDH-41184 IMPALA-3480">
+              <p>
+                New query options <codeph>RUNTIME_FILTER_MIN_SIZE</codeph> and
+                <codeph>RUNTIME_FILTER_MAX_SIZE</codeph> let you fine-tune
+                the sizes of the Bloom filter structures used for runtime filtering.
+                If the filter size derived from Impala internal estimates or from
+                the <codeph>RUNTIME_FILTER_BLOOM_SIZE</codeph> falls outside the size
+                range specified by these options, any too-small filter size is adjusted
+                to the minimum, and any too-large filter size is adjusted to the maximum.
+                See <xref href="impala_runtime_filter_min_size.xml#runtime_filter_min_size"/>
+                and <xref href="impala_runtime_filter_max_size.xml#runtime_filter_max_size"/>
+                for details about these options.
+              </p>
+            </li>
+            <li rev="CDH-41184 IMPALA-2956">
+              <p>
+                Runtime filter propagation now applies to all the
+                operands of <codeph>UNION</codeph> and <codeph>UNION ALL</codeph>
+                operators.
+              </p>
+            </li>
+            <li rev="CDH-41184 IMPALA-3077">
+              <p>
+                Runtime filters can now be produced during join queries even 
+                when the join processing activates the spill-to-disk mechanism.
+              </p>
+            </li>
+          </ul>
+            See <xref href="impala_runtime_filtering.xml#runtime_filtering"/> for
+            general information about the runtime filtering feature.
+        </li>
+        <!-- Have to look closer at resource management / admission control to see if
+             there are any ripple effects from this default change. -->
+        <li>
+          <p rev="IMPALA-3199">
+            Admission control and dynamic resource pools are enabled by default.
+            See <xref href="impala_admission.xml#admission_control"/> for details
+            about admission control.
+          </p>
+        </li>
+        <!-- Below here are features that are pretty well taken care of already;
+             some of them didn't need much if any doc in the first place. -->
+        <li>
+          <p rev="IMPALA-3369">
+            Impala can now manually set column statistics,
+            using the <codeph>ALTER TABLE</codeph> statement with a
+            <codeph>SET COLUMN STATS</codeph> clause.
+            See <xref href="impala_perf_stats.xml#perf_column_stats_manual"/> for details.
+          </p>
+        </li>
+        <li>
+          <p rev="CDH-40238 CDH-39818 IMPALA-3490 IMPALA-3581 IMPALA-2686">
+            Impala can now write lightweight <q>minidump</q> files, rather
+            than large core files, to save diagnostic information when
+            any of the Impala-related daemons crash. This feature uses the
+            open source <codeph>breakpad</codeph> framework.
+            See <xref href="impala_breakpad.xml#breakpad"/> for details.
+          </p>
+        </li>
+        <li>
+          <p>
+            New query options improve interoperability with Parquet files:
+            <ul>
+              <li>
+                <p rev="IMPALA-2835 CDH-33330">
+                  The <codeph>PARQUET_FALLBACK_SCHEMA_RESOLUTION</codeph> query option
+                  lets Impala locate columns within Parquet files based on
+                  column name rather than ordinal position.
+                  This enhancement improves interoperability with applications
+                  that write Parquet files with a different order or subset of
+                  columns than are used in the Impala table.
+                  See <xref href="impala_parquet_fallback_schema_resolution.xml#parquet_fallback_schema_resolution"/>
+                  for details.
+                </p>
+              </li>
+              <li>
+                <p rev="IMPALA-2069">
+                  The <codeph>PARQUET_ANNOTATE_STRINGS_UTF8</codeph> query option
+                  makes Impala include the <codeph>UTF-8</codeph> annotation
+                  metadata for <codeph>STRING</codeph>, <codeph>CHAR</codeph>,
+                  and <codeph>VARCHAR</codeph> columns in Parquet files created
+                  by <codeph>INSERT</codeph> or <codeph>CREATE TABLE AS SELECT</codeph>
+                  statements.
+                  See <xref href="impala_parquet_annotate_strings_utf8.xml#parquet_annotate_strings_utf8"/>
+                  for details.
+                </p>
+              </li>
+            </ul>
+            See <xref href="impala_parquet.xml#parquet"/> for general information about working
+            with Parquet files.
+          </p>
+        </li>
+        <li>
+          <p>
+            Improvements to security and reduction in overhead for secure clusters:
+          </p>
+          <ul>
+            <li>
+              <p rev="IMPALA-1928">
+                Overall performance improvements for secure clusters.
+                (TPC-H queries on a secure cluster were benchmarked
+                at roughly 3x as fast as the previous release.)
+              </p>
+            </li>
+            <li>
+              <p rev="IMPALA-2660 CDH-40241">
+                Impala now recognizes the <codeph>auth_to_local</codeph> setting,
+                specified through the HDFS configuration setting
+                <codeph>hadoop.security.auth_to_local</codeph>.
+                This feature is disabled by default; to enable it,
+                specify <codeph>--load_auth_to_local_rules=true</codeph>
+                in the <cmdname>impalad</cmdname> configuration settings.
+                See <xref href="impala_kerberos.xml#auth_to_local"/> for details.
+              </p>
+            </li>
+            <li>
+              <p rev="IMPALA-2599">
+                Timing improvements in the mechanism for the <cmdname>impalad</cmdname>
+                daemon to acquire Kerberos tickets. This feature spreads out the overhead
+                on the KDC during Impala startup, especially for large clusters.
+              </p>
+            </li>
+            <li>
+              <p rev="IMPALA-3554">
+                For Kerberized clusters, the Catalog service now uses
+                the Kerberos principal instead of the operating sytem user that runs
+                the <cmdname>catalogd</cmdname> daemon. 
+                This eliminates the requirement to configure a <codeph>hadoop.user.group.static.mapping.overrides</codeph> 
+                setting to put the OS user into the Sentry administrative group, on clusters where the principal
+                and the OS user name for this user are different.
+              </p>
+            </li>
+          </ul>
+        </li>
+        <li>
+          <p rev="IMPALA-3286">
+            Overall performance improvements for join queries, by using a prefetching mechanism
+            while building the in-memory hash table to evaluate join predicates.
+            See <xref href="impala_prefetch_mode.xml#prefetch_mode"/> for the query option
+            to control this optimization.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-3397 CDH-40097">
+            The <cmdname>impala-shell</cmdname> interpreter has a new command,
+            <codeph>SOURCE</codeph>, that lets you run a set of SQL statements
+            or other <cmdname>impala-shell</cmdname> commands stored in a file.
+            You can run additional <codeph>SOURCE</codeph> commands from inside
+            a file, to set up flexible sequences of statements for use cases
+            such as schema setup, ETL, or reporting.
+            See <xref href="impala_shell_commands.xml#shell_commands"/> for details
+            and <xref href="impala_shell_running_commands.xml#shell_running_commands"/>
+            for examples.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-1772 CDH-38381">
+            The <codeph>millisecond()</codeph> built-in function lets you extract
+            the fractional seconds part of a <codeph>TIMESTAMP</codeph> value.
+            See <xref href="impala_datetime_functions.xml#datetime_functions"/> for details.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-3092">
+            If an Avro table is created without column definitions in the
+            <codeph>CREATE TABLE</codeph> statement, and columns are later
+            added through <codeph>ALTER TABLE</codeph>, the resulting
+            table is now queryable. Missing values from the newly added
+            columns now default to <codeph>NULL</codeph>.
+            See <xref href="impala_avro.xml#avro"/> for general details about
+            working with Avro files.
+          </p>
+        </li>
+        <li>
+          <p>
+            The mechanism for interpreting <codeph>DECIMAL</codeph> literals is
+            improved, no longer going through an intermediate conversion step
+            to <codeph>DOUBLE</codeph>:
+            <ul>
+              <li>
+                <p rev="IMPALA-3163">
+                  Casting a <codeph>DECIMAL</codeph> value to <codeph>TIMESTAMP</codeph>
+                  <codeph>DOUBLE</codeph> produces a more precise
+                  value for the <codeph>TIMESTAMP</codeph> than formerly.
+                </p>
+              </li>
+              <li>
+                <p rev="IMPALA-3439">
+                  Certain function calls involving <codeph>DECIMAL</codeph> literals
+                  now succeed, when formerly they failed due to lack of a function
+                  signature with a <codeph>DOUBLE</codeph> argument.
+                </p>
+              </li>
+              <li>
+                <p rev="">
+                  Faster runtime performance for <codeph>DECIMAL</codeph> constant
+                  values, through improved native code generation for all combinations
+                  of precision and scale.
+                </p>
+              </li>
+            </ul>
+            See <xref href="impala_decimal.xml#decimal"/> for details about the <codeph>DECIMAL</codeph> type.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-3155">
+            Improved type accuracy for <codeph>CASE</codeph> return values.
+            If all <codeph>WHEN</codeph> clauses of the <codeph>CASE</codeph>
+            expression are of <codeph>CHAR</codeph> type, the final result
+            is also <codeph>CHAR</codeph> instead of being converted to
+            <codeph>STRING</codeph>.
+            See <xref href="impala_conditional_functions.xml#conditional_functions"/>
+            for details about the <codeph>CASE</codeph> function.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-3232">
+            Uncorrelated queries using the <codeph>NOT EXISTS</codeph> operator
+            are now supported. Formerly, the <codeph>NOT EXISTS</codeph>
+            operator was only available for correlated subqueries.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-2736">
+            Improved performance for reading Parquet files.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-3375">
+            Improved performance for <term>top-N</term> queries, that is,
+            those including both <codeph>ORDER BY</codeph> and
+            <codeph>LIMIT</codeph> clauses.
+          </p>
+        </li>
+        <!-- JIRA still in open state as of 5.8 / 2.6, commenting out.
+        <li>
+          <p rev="IMPALA-3471">
+            A top-N query can now also activate the spill-to-disk mechanism if
+            a host runs low on memory while evaluating it. For example, using
+            large <codeph>LIMIT</codeph> and/or <codeph>OFFSET</codeph> clauses
+            adds some memory overhead that could cause spilling.
+          </p>
+        </li>
+        -->
+        <li>
+          <p rev="IMPALA-1740">
+            Impala optionally skips an arbitrary number of header lines from text input
+            files on HDFS based on the <codeph>skip.header.line.count</codeph> value
+            in the <codeph>TBLPROPERTIES</codeph> field of the table metadata.
+            See <xref href="impala_txtfile.xml#text_data_files"/> for details.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-2336">
+            Trailing comments are now allowed in queries processed by
+            the <cmdname>impala-shell</cmdname> options <codeph>-q</codeph>
+            and <codeph>-f</codeph>.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-2844">
+            Impala can run <codeph>COUNT</codeph> queries for RCFile tables
+            that include complex type columns.
+            See <xref href="impala_complex_types.xml#complex_types"/> for
+            general information about working with complex types,
+            and <xref href="impala_array.xml#array"/>,
+            <xref href="impala_map.xml#map"/>, and <xref href="impala_struct.xml#struct"/>
+            for syntax details of each type.
+          </p>
+        </li>
+      </ul>
+
+    </conbody>
+  </concept>
+
+<!-- All 2.5.x new features go under here -->
+
+  <concept rev="2.5.0" id="new_features_250">
+
+    <title>New Features in Impala 2.5.x / CDH 5.7.x</title>
+
+    <conbody>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_250" />
+
+      <ul>
+        <li><!-- Spec: https://docs.google.com/document/d/1ambtYJ1t05iITCVIrN6N1A-e7PZBSetBPgjy8SLzJrA/edit#heading=h.vcftzwlpn845 -->
+          <p rev="CDH-33292 IMPALA-2552 IMPALA-3054">
+            Dynamic partition pruning. When a query refers to a partition key column in a <codeph>WHERE</codeph>
+            clause, and the exact set of column values are not known until the query is executed,
+            Impala evaluates the predicate and skips the I/O for entire partitions that are not needed.
+            For example, if a table was partitioned by year, Impala would apply this technique to a query
+            such as <codeph>SELECT c1 FROM partitioned_table WHERE year = (SELECT MAX(year) FROM other_table)</codeph>.
+            <ph audience="standalone">See <xref href="impala_partitioning.xml#dynamic_partition_pruning"/> for details.</ph>
+          </p>
+          <p>
+            The dynamic partition pruning optimization technique lets Impala avoid reading
+            data files from partitions that are not part of the result set, even when
+            that determination cannot be made in advance. This technique is especially valuable
+            when performing join queries involving partitioned tables. For example, if a join
+            query includes an <codeph>ON</codeph> clause and a <codeph>WHERE</codeph> clause
+            that refer to the same columns, the query can find the set of column values that
+            match the <codeph>WHERE</codeph> clause, and only scan the associated partitions
+            when evaluating the <codeph>ON</codeph> clause.
+          </p>
+          <p>
+            Dynamic partition pruning is controlled by the same settings as the runtime filtering feature.
+            By default, this feature is enabled at a medium level, because the maximum setting can use
+            slightly more memory for queries than in previous releases.
+            To fully enable this feature, set the query option <codeph>RUNTIME_FILTER_MODE=GLOBAL</codeph>.
+          </p>
+        </li>
+        <li><!-- Spec: https://docs.google.com/document/d/1ambtYJ1t05iITCVIrN6N1A-e7PZBSetBPgjy8SLzJrA/edit#heading=h.vcftzwlpn845 -->
+          <p rev="IMPALA-2419 IMPALA-3001 IMPALA-3008 IMPALA-3039 IMPALA-3046 IMPALA-3054">
+            Runtime filtering. This is a wide-ranging set of optimizations that are especially valuable for join queries.
+            Using the same technique as with dynamic partition pruning,
+            Impala uses the predicates from <codeph>WHERE</codeph> and <codeph>ON</codeph> clauses
+            to determine the subset of column values from one of the joined tables could possibly be part of the
+            result set. Impala sends a compact representation of the filter condition to the hosts in the cluster,
+            instead of the full set of values or the entire table.
+            <ph audience="PDF">See <xref href="impala_runtime_filtering.xml#runtime_filtering"/> for details.</ph>
+          </p>
+          <p>
+            By default, this feature is enabled at a medium level, because the maximum setting can use
+            slightly more memory for queries than in previous releases.
+            To fully enable this feature, set the query option <codeph>RUNTIME_FILTER_MODE=GLOBAL</codeph>.
+            <ph audience="PDF">See <xref href="impala_runtime_filter_mode.xml#runtime_filter_mode"/> for details.</ph>
+          </p>
+          <p>
+            This feature involves some new query options:
+            <xref audience="standalone" href="impala_runtime_filter_mode.xml">RUNTIME_FILTER_MODE</xref><codeph audience="integrated">RUNTIME_FILTER_MODE</codeph>,
+            <xref audience="standalone" href="impala_max_num_runtime_filters.xml">MAX_NUM_RUNTIME_FILTERS</xref><codeph audience="integrated">MAX_NUM_RUNTIME_FILTERS</codeph>,
+            <xref audience="standalone" href="impala_runtime_bloom_filter_size.xml">RUNTIME_BLOOM_FILTER_SIZE</xref><codeph audience="integrated">RUNTIME_BLOOM_FILTER_SIZE</codeph>,
+            <xref audience="standalone" href="impala_runtime_filter_wait_time_ms.xml">RUNTIME_FILTER_WAIT_TIME_MS</xref><codeph audience="integrated">RUNTIME_FILTER_WAIT_TIME_MS</codeph>,
+            and <xref audience="standalone" href="impala_disable_row_runtime_filtering.xml">DISABLE_ROW_RUNTIME_FILTERING</xref><codeph audience="integrated">DISABLE_ROW_RUNTIME_FILTERING</codeph>.
+            <ph audience="PDF">See
+            <xref href="impala_runtime_filter_mode.xml#runtime_filter_mode">RUNTIME_FILTER_MODE</xref>,
+            <xref href="impala_max_num_runtime_filters.xml#max_num_runtime_filters">MAX_NUM_RUNTIME_FILTERS</xref>,
+            <xref href="impala_runtime_bloom_filter_size.xml#runtime_bloom_filter_size">RUNTIME_BLOOM_FILTER_SIZE</xref>,
+            <xref href="impala_runtime_filter_wait_time_ms.xml#runtime_filter_wait_time_ms">RUNTIME_FILTER_WAIT_TIME_MS</xref>, and
+            <xref href="impala_disable_row_runtime_filtering.xml#disable_row_runtime_filtering">DISABLE_ROW_RUNTIME_FILTERING</xref>
+            for details.
+            </ph>
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-2696">
+            More efficient use of the HDFS caching feature, to avoid
+            hotspots and bottlenecks that could occur if heavily used
+            cached data blocks were always processed by the same host.
+            By default, Impala now randomizes which host processes each cached
+            HDFS data block, when cached replicas are available on multiple hosts.
+            (Remember to use the <codeph>WITH REPLICATION</codeph> clause with the
+            <codeph>CREATE TABLE</codeph> or <codeph>ALTER TABLE</codeph> statement
+            when enabling HDFS caching for a table or partition, to cache the same
+            data blocks across multiple hosts.)
+            The new query option <codeph>SCHEDULE_RANDOM_REPLICA</codeph>
+            <!-- and <codeph>REPLICA_PREFERENCE</codeph> -->
+            lets you fine-tune the interaction with HDFS caching even more.
+            <ph audience="PDF">See <xref href="impala_perf_hdfs_caching.xml#hdfs_caching"/> for details.</ph>
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-2641">
+            The <codeph>TRUNCATE TABLE</codeph> statement now accepts an <codeph>IF EXISTS</codeph>
+            clause, making <codeph>TRUNCATE TABLE</codeph> easier to use in setup or ETL scripts where the table might or
+            might not exist.
+            <ph audience="PDF">See <xref href="impala_truncate_table.xml#truncate_table"/> for details.</ph>
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-2681 IMPALA-2688 IMPALA-2749">
+            Improved performance and reliability for the <codeph>DECIMAL</codeph> data type:
+            <ul>
+            <li>
+              <p rev="IMPALA-2681">
+                Using <codeph>DECIMAL</codeph> values in a <codeph>GROUP BY</codeph> clause now
+                triggers the native code generation optimization, speeding up queries that
+                group by values such as prices.
+              </p>
+            </li>
+            <li>
+              <p rev="IMPALA-2688">
+                Checking for overflow in <codeph>DECIMAL</codeph>
+                multiplication is now substantially faster, making <codeph>DECIMAL</codeph>
+                a more practical data type in some use cases where formerly <codeph>DECIMAL</codeph>
+                was much slower than <codeph>FLOAT</codeph> or <codeph>DOUBLE</codeph>.
+              </p>
+            </li>
+            <li>
+              <p rev="IMPALA-2749">
+                Multiplying a mixture of <codeph>DECIMAL</codeph>
+                and <codeph>FLOAT</codeph> or <codeph>DOUBLE</codeph> values now returns the
+                <codeph>DOUBLE</codeph> rather than <codeph>DECIMAL</codeph>. This change avoids
+                some cases where an intermediate value would underflow or overflow and become
+                <codeph>NULL</codeph> unexpectedly.
+              </p>
+            </li>
+            </ul>
+            <ph audience="PDF">See <xref href="impala_decimal.xml"/> for details.</ph>
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-2382">
+            For UDFs written in Java, or Hive UDFs reused for Impala,
+            Impala now allows parameters and return values to be primitive types.
+            Formerly, these things were required to be one of the <q>Writable</q>
+            object types.
+            <ph audience="PDF">See <xref href="impala_udf.xml#udfs_hive"/> for details.</ph>
+          </p>
+        </li>
+        <!-- CDH-33298 is for scoping internationalization / UTF-8 / Unicode support. That work is pushed out to 5.8.
+        <li>
+          <p rev="CDH-33298">
+            Improvements to internationalization support.
+            Now Impala can process data that uses the UTF-8 character encoding.
+          </p>
+        </li>
+        -->
+        <li>
+          <p rev="IMPALA-1588"><!-- This is from 2015, so perhaps it's really in an earlier release. -->
+            Performance improvements for HDFS I/O. Impala now caches HDFS file handles to avoid the
+            overhead of repeatedly opening the same file.
+          </p>
+        </li>
+
+        <!-- Kudu didn't make it into 2.5 / 5.7 release, so no DELETE or UPDATE statement. -->
+        <li>
+          <p><!-- Is there a JIRA for that one? Alex? -->
+            Performance improvements for queries involving nested complex types.
+            Certain basic query types, such as counting the elements of a complex column,
+            now use an optimized code path.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-3044 IMPALA-2538 IMPALA-1168 CDH-33289 CDH-34603">
+            Improvements to the memory reservation mechanism for the Impala
+            admission control feature. You can specify more settings, such
+            as the timeout period and maximum aggregate memory used, for each
+            resource pool instead of globally for the Impala instance. The
+            default limit for concurrent queries (the <uicontrol>max requests</uicontrol>
+            setting) is now unlimited instead of 200.
+            The Cloudera Manager user interface for admission control has been
+            reworked, with the settings available under the
+            <uicontrol>Dynamic Resource Pools</uicontrol> window.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-1755">
+            Performance improvements related to code generation.
+            Even in queries where code generation is not performed
+            for some phases of execution (such as reading data from
+            Parquet tables), Impala can still use code generation in
+            other parts of the query, such as evaluating
+            functions in the <codeph>WHERE</codeph> clause.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-1305">
+            Performance improvements for queries using aggregation functions
+            on high-cardinality columns.
+            Formerly, Impala could do unnecessary extra work to produce intermediate
+            results for operations such as <codeph>DISTINCT</codeph> or <codeph>GROUP BY</codeph>
+            on columns that were unique or had few duplicate values.
+            Now, Impala decides at run time whether it is more efficient to
+            do an initial aggregation phase and pass along a smaller set of intermediate data,
+            or to pass raw intermediate data back to next phase of query processing to be aggregated there.
+            This feature is known as <term>streaming pre-aggregation</term>.
+            In case of performance regression, this feature can be turned off
+            using the <codeph>DISABLE_STREAMING_PREAGGREGATIONS</codeph> query option.
+            <ph audience="PDF">See <xref href="impala_disable_streaming_preaggregations.xml#disable_streaming_preaggregations"/> for details.</ph>
+          </p>
+        </li>
+        <li>
+          <p>
+            Spill-to-disk feature now always recommended. In earlier releases, the spill-to-disk feature
+            could be turned off using a pair of configuration settings,
+            <codeph>enable_partitioned_aggregation=false</codeph> and
+            <codeph>enable_partitioned_hash_join=false</codeph>.
+            The latest improvements in the spill-to-disk mechanism, and related features that
+            interact with it, make this feature robust enough that disabling it is now
+            no longer needed or supported. In particular, some new features in <keyword keyref="impala25_full"/>
+            and higher do not work when the spill-to-disk feature is disabled.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-1067">
+            Improvements to scripting capability for the <cmdname>impala-shell</cmdname> command,
+            through user-specified substitution variables that can appear in statements processed
+            by <cmdname>impala-shell</cmdname>:
+          </p>
+          <ul>
+            <li rev="IMPALA-2179">
+              <p>
+                The <codeph>--var</codeph> command-line option lets you pass key-value pairs to
+                <cmdname>impala-shell</cmdname>. The shell can substitute the values
+                into queries before executing them, where the query text contains the notation
+                <codeph>${var:<varname>varname</varname>}</codeph>. For example, you might prepare a SQL file
+                containing a set of DDL statements and queries containing variables for
+                database and table names, and then pass the applicable names as part of the
+                <codeph>impala-shell -f <varname>filename</varname></codeph> command.
+                <ph audience="PDF">See <xref href="impala_shell_running_commands.xml#shell_running_commands"/> for details.</ph>
+              </p>
+            </li>
+            <li rev="IMPALA-2180">
+              <p>
+                The <codeph>SET</codeph> and <codeph>UNSET</codeph> commands within the
+                <cmdname>impala-shell</cmdname> interpreter now work with user-specified
+                substitution variables, as well as the built-in query options.
+                The two kinds of variables are divided in the <codeph>SET</codeph> output.
+                As with variables defined by the <codeph>--var</codeph> command-line option,
+                you refer to the user-specified substitution variables in queries by using
+                the notation <codeph>${var:<varname>varname</varname>}</codeph>
+                in the query text. Because the substitution variables are processed by
+                <cmdname>impala-shell</cmdname> instead of the <cmdname>impalad</cmdname>
+                backend, you cannot define your own substitution variables through the
+                <codeph>SET</codeph> statement in a JDBC or ODBC application.
+                <ph audience="PDF">See <xref href="impala_set.xml#set"/> for details.</ph>
+              </p>
+            </li>
+          </ul>
+        </li>
+        <li>
+          <p rev="IMPALA-1599">
+            Performance improvements for query startup. Impala better parallelizes certain work
+            when coordinating plan distribution between <cmdname>impalad</cmdname> instances, which improves
+            startup time for queries involving tables with many partitions on large clusters,
+            or complicated queries with many plan fragments.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-2560">
+            Performance and scalability improvements for tables with many partitions.
+            The memory requirements on the coordinator node are reduced, making it substantially
+            faster and less resource-intensive
+            to do joins involving several tables with thousands of partitions each.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-3095">
+            Whitelisting for access to internal APIs. For applications that need direct access
+            to Impala APIs, without going through the HiveServer2 or Beeswax interfaces, you can
+            specify a list of Kerberos users who are allowed to call those APIs. By default, the
+            <codeph>impala</codeph> and <codeph>hdfs</codeph> users are the only ones authorized
+            for this kind of access.
+            Any users not explicitly authorized through the <codeph>internal_principals_whitelist</codeph>
+            configuration setting are blocked from accessing the APIs. This setting applies to all the
+            Impala-related daemons, although currently it is primarily used for HDFS to control the
+            behavior of the catalog server.
+          </p>
+        </li>
+        <li>
+          <p rev="CDH-37009 CDH-30378">
+            Improvements to Impala integration and usability for Hue. (The code changes
+            are actually on the Hue side.)
+          </p>
+          <ul>
+          <li>
+            <p rev="CDH-37009">
+              The list of tables now refreshes dynamically.
+            </p>
+          </li>
+          </ul>
+        </li>
+        <li>
+          <p rev="IMPALA-1787">
+            Usability improvements for case-insensitive queries.
+            You can now use the operators <codeph>ILIKE</codeph> and <codeph>IREGEXP</codeph>
+            to perform case-insensitive wildcard matches or regular expression matches,
+            rather than explicitly converting column values with <codeph>UPPER</codeph>
+            or <codeph>LOWER</codeph>.
+            <ph audience="PDF">See <xref href="impala_operators.xml#ilike"/> and <xref href="impala_operators.xml#iregexp"/> for details.</ph>
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-1480">
+            Performance and reliability improvements for DDL and insert operations on partitioned tables with a large
+            number of partitions. Impala only re-evaluates metadata for partitions that are affected by
+            a DDL operation, not all partitions in the table. While a DDL or insert statement is in progress,
+            other Impala statements that attempt to modify metadata for the same table wait until the first one
+            finishes.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-2867">
+            Reliability improvements for the <codeph>LOAD DATA</codeph> statement.
+            Previously, this statement would fail if the source HDFS directory
+            contained any subdirectories at all. Now, the statement ignores
+            any hidden subdirectories, for example <filepath>_impala_insert_staging</filepath>.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-2147">
+            A new operator, <codeph>IS [NOT] DISTINCT FROM</codeph>, lets you compare values
+            and always get a <codeph>true</codeph> or <codeph>false</codeph> result,
+            even if one or both of the values are <codeph>NULL</codeph>.
+            The <codeph>IS NOT DISTINCT FROM</codeph> operator, or its equivalent
+            <codeph>&lt;=&gt;</codeph> notation, improves the efficiency of join queries that
+            treat key values that are <codeph>NULL</codeph> in both tables as equal.
+            <ph audience="PDF">See <xref href="impala_operators.xml#is_distinct_from"/> for details.</ph>
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-1934">
+            Security enhancements for the <cmdname>impala-shell</cmdname> command.
+            A new option, <codeph>--ldap_password_cmd</codeph>, lets you specify
+            a command to retrieve the LDAP password. The resulting password is
+            then used to authenticate the <cmdname>impala-shell</cmdname> command
+            with the LDAP server.
+            <ph audience="PDF">See <xref href="impala_shell_options.xml"/> for details.</ph>
+          </p>
+        </li>
+        <li>
+          <p>
+            The <codeph>CREATE TABLE AS SELECT</codeph> statement now accepts a
+            <codeph>PARTITIONED BY</codeph> clause, which lets you create a
+            partitioned table and insert data into it with a single statement.
+            <ph audience="PDF">See <xref href="impala_create_table.xml#create_table"/> for details.</ph>
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-1748 CDH-38369">
+            User-defined functions (UDFs and UDAFs) written in C++ now persist automatically
+            when the <cmdname>catalogd</cmdname> daemon is restarted. You no longer
+            have to run the <codeph>CREATE FUNCTION</codeph> statements again after a restart.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-2843 CDH-39148">
+            User-defined functions (UDFs) written in Java can now persist
+            when the <cmdname>catalogd</cmdname> daemon is restarted, and can be shared
+            transparently between Impala and Hive. You must do a one-time operation to recreate these
+            UDFs using new <codeph>CREATE FUNCTION</codeph> syntax, without a signature for arguments
+            or the return value. Afterwards, you no longer have to run the <codeph>CREATE FUNCTION</codeph>
+            statements again after a restart.
+            Although Impala does not have visibility into the UDFs that implement the 
+            Hive built-in functions, user-created Hive UDFs are now automatically available
+            for calling through Impala.
+            <ph audience="PDF">See <xref href="impala_create_function.xml#create_function"/> for details.</ph>
+          </p>
+        </li>
+        <li>
+          <!-- Listed as fixed in 2.6.0. Is this item inappropriate or did it actually come from a different JIRA? -->
+          <p rev="IMPALA-2728">
+            Reliability enhancements for memory management. Some aggregation and join queries
+            that formerly might have failed with an out-of-memory error due to memory contention,
+            now can succeed using the spill-to-disk mechanism.
+          </p>
+        </li>
+        <li>
+          <!-- Same blurb is under Incompatible Changes. Turn into a conref. -->
+          <p rev="IMPALA-2070">
+            The <codeph>SHOW DATABASES</codeph> statement now returns two columns rather than one.
+            The second column includes the associated comment string, if any, for each database.
+            Adjust any application code that examines the list of databases and assumes the
+            result set contains only a single column.
+            <ph audience="PDF">See <xref href="impala_show.xml#show_databases"/> for details.</ph>
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-2499">
+            A new optimization speeds up aggregation operations that involve only the partition key
+            columns of partitioned tables. For example, a query such as <codeph>SELECT COUNT(DISTINCT k), MIN(k), MAX(k) FROM t1</codeph>
+            can avoid reading any data files if <codeph>T1</codeph> is a partitioned table and <codeph>K</codeph>
+            is one of the partition key columns. Because this technique can produce different results in cases
+            where HDFS files in a partition are manually deleted or are empty, you must enable the optimization
+            by setting the query option <codeph>OPTIMIZE_PARTITION_KEY_SCANS</codeph>.
+            <ph audience="PDF">See <xref href="impala_optimize_partition_key_scans.xml"/> for details.</ph>
+          </p>
+        </li>
+        <li audience="Cloudera"><!-- All the other undocumented query options are not really new features for this release, so hiding this whole bullet. -->
+          <p>
+            Other new query options:
+          </p>
+          <ul>
+            <li audience="Cloudera"><!-- Actually from a long way back, just never documented. Not sure if appropriate to keep internal-only or expose. -->
+              <codeph>DISABLE_OUTERMOST_TOPN</codeph>
+            </li>
+            <li audience="Cloudera"><!-- Actually from a long way back, just never documented. Not sure if appropriate to keep internal-only or expose. -->
+              <codeph>RM_INITIAL_MEM</codeph>
+            </li>
+            <li audience="Cloudera"><!-- Seems to be related to writing sequence files, a capability not externalized at this time. -->
+              <codeph>SEQ_COMPRESSION_MODE</codeph>
+            </li>
+            <li audience="Cloudera"><!-- Actually, was only used for working around one JIRA. Being deprecated now in Impala 2.3 via IMPALA-2963. -->
+              <codeph>DISABLE_CACHED_READS</codeph>
+            </li>
+          </ul>
+        </li>
+        <li>
+          <p rev="IMPALA-2196">
+            The <codeph>DESCRIBE</codeph> statement can now display metadata about a database, using the
+            syntax <codeph>DESCRIBE DATABASE <varname>db_name</varname></codeph>.
+            <ph audience="PDF">See <xref href="impala_describe.xml#describe"/> for details.</ph>
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-1477">
+            The <codeph>uuid()</codeph> built-in function generates an
+            alphanumeric value that you can use as a guaranteed unique identifier.
+            The uniqueness applies even across tables, for cases where an ascending
+            numeric sequence is not suitable.
+            <ph audience="PDF">See <xref href="impala_misc_functions.xml#misc_functions"/> for details.</ph>
+          </p>
+        </li>
+      </ul>
+
+    </conbody>
+  </concept>
+
+<!-- All 2.4.x new features go under here -->
+
+  <concept rev="2.4.0" id="new_features_240">
+
+    <title>New Features in Impala 2.4.x / CDH 5.6.x</title>
+
+    <conbody>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_240" />
+
+      <ul>
+        <li>
+          <p>
+            Impala can be used on the DSSD D5 Storage Appliance.
+            From a user perspective, the Impala features are the same as in CDH 5.5 / Impala 2.3.
+          </p>
+        </li>
+      </ul>
+
+    </conbody>
+  </concept>
+
+<!-- All 2.3.x subsections go under here -->
+
+<!-- Actually for 2.3 / 5.5, let's get away from doing a separate subhead for each maintenance release,
+     because in the normal course of events there will be nothing to add here until 5.6. If something new
+     needs to get noted, just add a new bullet with wording to indicate which 5.5.x release it applies to. -->
+
+  <concept rev="2.3.0" id="new_features_230">
+
+    <title>New Features in Impala 2.3.x / CDH 5.5.x</title>
+
+    <conbody>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_23x" />
+
+      <p>
+        The following are the major new features in Impala 2.3.x. This major release, available as part of CDH
+        5.5.x, contains improvements to SQL syntax (particularly new support for complex types), performance,
+        manageability, security.
+      </p>
+
+      <ul>
+
+        <li>
+          <p>
+            Complex data types: <codeph>STRUCT</codeph>, <codeph>ARRAY</codeph>, and <codeph>MAP</codeph>. These
+            types can encode multiple named fields, positional items, or key-value pairs within a single column.
+            You can combine these types to produce nested types with arbitrarily deep nesting,
+            such as an <codeph>ARRAY</codeph> of <codeph>STRUCT</codeph> values,
+            a <codeph>MAP</codeph> where each key-value pair is an <codeph>ARRAY</codeph> of other <codeph>MAP</codeph> values,
+            and so on. Currently, complex data types are only supported for the Parquet file format.
+            <ph audience="PDF">See <xref href="impala_complex_types.xml#complex_types"/> for usage details and <xref href="impala_array.xml#array"/>, <xref href="impala_struct.xml#struct"/>, and <xref href="impala_map.xml#map"/> for syntax.</ph>
+          </p>
+        </li>
+
+        <li rev="collevelauth">
+          <p>
+            Column-level authorization lets you define access to particular columns within a table,
+            rather than the entire table. This feature lets you reduce the reliance on creating views to
+            set up authorization schemes for subsets of information.
+            <ph audience="integrated">See <xref href="sg_hive_sql.xml#concept_c2q_4qx_p4/col_level_auth_sentry"/> for background details, and <xref href="impala_grant.xml#grant"/> and <xref href="impala_revoke.xml#revoke"/> for Impala-specific syntax.</ph>
+          </p>
+        </li>
+
+        <li rev="IMPALA-1139">
+          <p>
+            The <codeph>TRUNCATE TABLE</codeph> statement removes all the data from a table without removing the table itself.
+            <ph audience="PDF">See <xref href="impala_truncate_table.xml#truncate_table"/> for details.</ph>
+          </p>
+        </li>
+
+        <li id="IMPALA-2015">
+          <p>
+            Nested loop join queries. Some join queries that formerly required equality comparisons can now use
+            operators such as <codeph>&lt;</codeph> or <codeph>&gt;=</codeph>. This same join mechanism is used
+            internally to optimize queries that retrieve values from complex type columns.
+            <ph audience="PDF">See <xref href="impala_joins.xml#joins"/> for details about Impala join queries.</ph>
+          </p>
+        </li>
+
+        <li id="CDH-28141">
+          <p>
+            Reduced memory usage and improved performance and robustness for spill-to-disk feature.
+            <ph audience="PDF">See <xref href="impala_scalability.xml#spill_to_disk"/> for details about this feature.</ph>
+          </p>
+        </li>
+
+        <li rev="IMPALA-1881 CDH-34620">
+          <p>
+            Performance improvements for querying Parquet data files containing multiple row groups
+            and multiple data blocks:
+          </p>
+          <ul>
+          <li>
+          <p> For files written by Hive, SparkSQL, and other Parquet MR writers
+                and spanning multiple HDFS blocks, Impala now scans the extra
+                data blocks locally when possible, rather than using remote
+                reads. </p>
+          </li>
+          <li>
+          <p>
+            Impala queries benefit from the improved alignment of row groups with HDFS blocks for Parquet
+            files written by Hive, MapReduce, and other components in <ph rev="upstream">CDH 5.5</ph> and higher. (Impala itself never writes
+            multiblock Parquet files, so the alignment change does not apply to Parquet files produced by Impala.)
+            These Parquet writers now add padding to Parquet files that they write to align row groups with HDFS blocks.
+            The <codeph>parquet.writer.max-padding</codeph> setting specifies the maximum number of bytes, by default
+            8 megabytes, that can be added to the file between row groups to fill the gap at the end of one block
+            so that the next row group starts at the beginning of the next block.
+            If the gap is larger than this size, the writer attempts to fit another entire row group in the remaining space.
+            Include this setting in the <filepath>hive-site</filepath> configuration file to influence Parquet files written by Hive,
+            or the <filepath>hdfs-site</filepath> configuration file to influence Parquet files written by all non-Impala components.
+          </p>
+          </li>
+          </ul>
+          <p audience="PDF">
+            See <xref href="impala_parquet.xml#parquet"/> for instructions about using Parquet data files
+            with Impala, and
+            <xref audience="integrated" href="cdh_ig_parquet.xml#parquet_format"/><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_parquet.html" scope="external" format="html"/>
+            for instructions for
+            other components that can read and write Parquet files.
+          </p>
+        </li>
+
+        <li id="IMPALA-1660">
+          <p>
+            Many new built-in scalar functions, for convenience and enhanced portability of SQL that uses common industry extensions.
+          </p>
+
+          <p rev="IMPALA-1771">
+            Math functions<ph audience="PDF"> (see <xref href="impala_math_functions.xml#math_functions"/> for details)</ph>:
+          </p>
+          <ul>
+            <li>
+              <codeph>ATAN2</codeph>
+            </li>
+
+            <li>
+              <codeph>COSH</codeph>
+            </li>
+
+            <li>
+              <codeph>COT</codeph>
+            </li>
+
+            <li>
+              <codeph>DCEIL</codeph>
+            </li>
+
+            <li>
+              <codeph>DEXP</codeph>
+            </li>
+
+            <li>
+              <codeph>DFLOOR</codeph>
+            </li>
+
+            <li>
+              <codeph>DLOG10</codeph>
+            </li>
+
+            <li>
+              <codeph>DPOW</codeph>
+            </li>
+
+            <li>
+              <codeph>DROUND</codeph>
+            </li>
+
+            <li>
+              <codeph>DSQRT</codeph>
+            </li>
+
+            <li>
+              <codeph>DTRUNC</codeph>
+            </li>
+
+            <li>
+              <codeph>FACTORIAL</codeph>, and corresponding <codeph>!</codeph> operator
+            </li>
+
+            <li>
+              <codeph>FPOW</codeph>
+            </li>
+
+            <li>
+              <codeph>RADIANS</codeph>
+            </li>
+
+            <li>
+              <codeph>RANDOM</codeph>
+            </li>
+
+            <li>
+              <codeph>SINH</codeph>
+            </li>
+
+            <li>
+              <codeph>TANH</codeph>
+            </li>
+          </ul>
+
+          <p>
+            String functions<ph audience="PDF"> (see <xref href="impala_string_functions.xml#string_functions"/> for details)</ph>:
+          </p>
+          <ul>
+            <li>
+              <codeph>BTRIM</codeph>
+            </li>
+            <li>
+              <codeph>CHR</codeph>
+            </li>
+            <li>
+              <codeph>REGEXP_LIKE</codeph>
+            </li>
+            <li>
+              <codeph>SPLIT_PART</codeph>
+            </li>
+          </ul>
+
+          <p>
+            Date and time functions<ph audience="PDF"> (see <xref href="impala_datetime_functions.xml#datetime_functions"/> for details)</ph>:
+          </p>
+          <ul>
+              <li>
+                <codeph>INT_MONTHS_BETWEEN</codeph>
+              </li>
+              <li>
+                <codeph>MONTHS_BETWEEN</codeph>
+              </li>
+              <li>
+                <codeph>TIMEOFDAY</codeph>
+              </li>
+              <li>
+                <codeph>TIMESTAMP_CMP</codeph>
+              </li>
+          </ul>
+
+          <p>
+            Bit manipulation functions<ph audience="PDF"> (see <xref href="impala_bit_functions.xml#bit_functions"/> for details)</ph>:
+          </p>
+          <ul>
+            <li>
+              <codeph>BITAND</codeph>
+            </li>
+
+            <li>
+              <codeph>BITNOT</codeph>
+            </li>
+
+            <li>
+              <codeph>BITOR</codeph>
+            </li>
+
+            <li>
+              <codeph>BITXOR</codeph>
+            </li>
+
+            <li>
+              <codeph>COUNTSET</codeph>
+            </li>
+
+            <li>
+              <codeph>GETBIT</codeph>
+            </li>
+
+            <li>
+              <codeph>ROTATELEFT</codeph>
+            </li>
+
+            <li>
+              <codeph>ROTATERIGHT</codeph>
+            </li>
+
+            <li>
+              <codeph>SETBIT</codeph>
+            </li>
+
+            <li>
+              <codeph>SHIFTLEFT</codeph>
+            </li>
+
+            <li>
+              <codeph>SHIFTRIGHT</codeph>
+            </li>
+          </ul>
+          <p>
+            Type conversion functions<ph audience="PDF"> (see <xref href="impala_conversion_functions.xml#conversion_functions"/> for details)</ph>:
+          </p>
+          <ul>
+            <li>
+              <codeph>TYPEOF</codeph>
+            </li>
+          </ul>
+          <p>
+            The <codeph>effective_user()</codeph> function<ph audience="PDF"> (see <xref href="impala_misc_functions.xml#misc_functions"/> for details)</ph>.
+          </p>
+        </li>
+
+        <li id="IMPALA-2081">
+          <p>
+            New built-in analytic functions: <codeph>PERCENT_RANK</codeph>, <codeph>NTILE</codeph>,
+            <codeph>CUME_DIST</codeph>.
+            <ph audience="PDF">See <xref href="impala_analytic_functions.xml#analytic_functions"/> for details.</ph>
+          </p>
+        </li>
+
+        <li id="IMPALA-595">
+          <p>
+            The <codeph>DROP DATABASE</codeph> statement now works for a non-empty database.
+            When you specify the optional <codeph>CASCADE</codeph> clause, any tables in the
+            database are dropped before the database itself is removed.
+            <ph audience="PDF">See <xref href="impala_drop_database.xml#drop_database"/> for details.</ph>
+          </p>
+        </li>
+
+        <li>
+          <p>
+            The <codeph>DROP TABLE</codeph> and <codeph>ALTER TABLE DROP PARTITION</codeph> statements have a new optional keyword, <codeph>PURGE</codeph>.
+            This keyword causes Impala to immediately remove the relevant HDFS data files rather than sending them to the HDFS trashcan.
+            This feature can help to avoid out-of-space errors on storage devices, and to avoid files being left behind in case of
+            a problem with the HDFS trashcan, such as the trashcan not being configured or being in a different HDFS encryption zone
+            than the data files.
+            <ph audience="PDF">See <xref href="impala_drop_table.xml#drop_table"/> and <xref href="impala_alter_table.xml#alter_table"/> for syntax.</ph>
+          </p>
+        </li>
+
+        <li id="IMPALA-80">
+          <p>
+            The <cmdname>impala-shell</cmdname> command has a new feature for live progress reporting. This feature
+            is enabled through the <codeph>--live_progress</codeph> and <codeph>--live_summary</codeph>
+            command-line options, or during a session through the <codeph>LIVE_SUMMARY</codeph> and
+            <codeph>LIVE_PROGRESS</codeph> query options.
+            <ph audience="PDF">See <xref href="impala_live_progress.xml#live_progress"/> and <xref href="impala_live_summary.xml#live_summary"/> for details.</ph>
+          </p>
+        </li>
+
+        <li>
+          <p>
+            The <cmdname>impala-shell</cmdname> command also now displays a random <q>tip of the day</q> when it starts.
+          </p>
+        </li>
+
+        <li id="IMPALA-1413">
+          <p>
+            The <cmdname>impala-shell</cmdname> option <codeph>-f</codeph> now recognizes a special filename
+            <codeph>-</codeph> to accept input from stdin.
+            <ph audience="PDF">See <xref href="impala_shell_options.xml#shell_options"/> for details about the options for running <cmdname>impala-shell</cmdname> in non-interactive mode.</ph>
+          </p>
+        </li>
+
+        <li id="IMPALA-1963">
+          <p>
+            Format strings for the <codeph>unix_timestamp()</codeph> function can now include numeric timezone offsets.
+            <ph audience="PDF">See <xref href="impala_datetime_functions.xml#datetime_functions"/> for details.</ph>
+          </p>
+        </li>
+
+        <li id="CDH-27547">
+          <p>
+            Impala can now run a specified command to obtain the password to decrypt a private-key PEM file,
+            rather than having the private-key file be unencrypted on disk.
+            <ph audience="PDF">See <xref href="impala_ssl.xml#ssl"/> for details.</ph>
+          </p>
+        </li>
+
+        <li id="IMPALA-859">
+          <p>
+            Impala components now can use SSL for more of their internal communication. SSL is used for
+            communication between all three Impala-related daemons when the configuration option
+            <codeph>ssl_server_certificate</codeph> is enabled. SSL is used for communication with client
+            applications when the configuration option <codeph>ssl_client_ca_certificate</codeph> is enabled.
+            <ph audience="PDF">See <xref href="impala_ssl.xml#ssl"/> for details.</ph>
+          </p>
+          <p>
+            Currently, you can only use one of server-to-server TLS/SSL encryption or Kerberos authentication.
+            This limitation is tracked by the issue
+            <xref href="https://issues.cloudera.org/browse/IMPALA-2598" scope="external" format="html">IMPALA-2598</xref>.
+          </p>
+        </li>
+
+        <li id="IMPALA-1829">
+          <p>
+            Improved flexibility for intermediate data types in user-defined aggregate functions (UDAFs).
+            <ph audience="PDF">See <xref href="impala_udf.xml#udafs"/> for details.</ph>
+          </p>
+        </li>
+
+      </ul>
+
+      <p>
+        In CDH 5.5.2 / Impala 2.3.2, the bug fix for <xref href="https://issues.cloudera.org/browse/IMPALA-2598" scope="external" format="html">IMPALA-2598</xref>
+        removes the restriction on using both Kerberos and SSL for internal communication between Impala components.
+      </p>
+
+<!-- End of new feature list for 2.3 / 5.5. -->
+
+    </conbody>
+
+  </concept>
+
+<!-- All 2.2.x subsections go under here -->
+
+<!-- Removing all the 5.4.x release subtopics for which there wasn't anything to say.
+     Same convention as used in 5.5.x, 5.6.x, 5.7.x, 5.8.x. Only have one subtopic for
+     the .0.
+  <concept rev="5.4.10" id="new_features_2210">
+
+    <title>New Features in Impala 2.2.10 / CDH 5.4.10</title>
+
+    <conbody>
+
+      <p>
+        No new features. This point release is exclusively a bug fix release.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_22x" />
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="5.4.9" id="new_features_229">
+
+    <title>New Features in Impala 2.2.9 / CDH 5.4.9</title>
+
+    <conbody>
+
+      <p>
+        No new features. This point release is exclusively a bug fix release.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_22x" />
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="5.4.8" id="new_features_228">
+
+    <title>New Features in Impala 2.2.8 / CDH 5.4.8</title>
+
+    <conbody>
+
+      <p>
+        No new features. This point release is exclusively a bug fix release.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_22x" />
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="5.4.7" id="new_features_227">
+
+    <title>New Features in Impala 2.2.7 / CDH 5.4.7</title>
+
+    <conbody>
+
+      <p>
+        No new features. This point release is exclusively a bug fix release.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_22x" />
+
+    </conbody>
+
+  </concept>
+
+  <concept audience="Cloudera" rev="5.4.6" id="new_features_226">
+
+    <title>New Features in Impala 2.2.6 / CDH 5.4.6</title>
+
+    <conbody>
+
+      <p>
+        No new features. This point release is exclusively a bug fix release.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_22x" />
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="5.4.5" id="new_features_225">
+
+    <title>New Features in Impala 2.2.x for CDH 5.4.5</title>
+
+    <conbody>
+
+      <p>
+        No new features. This point release is exclusively a bug fix release.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_22x" />
+
+    </conbody>
+
+  </concept>
+-->
+
+  <concept rev="5.4.3" id="new_features_223">
+
+    <title>New Features in Impala 2.2.x for CDH 5.4.3 and 5.4.4</title>
+
+    <conbody>
+
+      <p>
+        No new features added to the Impala code. The certification of Impala with EMC Isilon under CDH 5.4.4 means
+        that now you can query data stored on Isilon storage devices through Impala. See
+        <xref audience="integrated" href="cm_mc_isilon_service.xml"/><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_isilon_service.html" scope="external" format="html"/>
+        for details. The same level of Impala is included with both CDH
+        5.4.3 and 5.4.4.
+<!-- This point release is exclusively a bug fix release. -->
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_22x" />
+
+    </conbody>
+
+  </concept>
+
+<!-- I let the 5.4.3/5.4.3 subtopic above remain in existence, but now back to hiding specific 5.4.x subtopics
+     after the .0 one that has the actual new features.
+  <concept audience="Cloudera" rev="5.4.2" id="new_features_222">
+
+    <title>New Features in Impala 2.2.x for CDH 5.4.2</title>
+
+    <conbody>
+
+      <p>
+        No new features. This point release is exclusively a bug fix release.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_22x" />
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="5.4.x" id="new_features_54x">
+
+    <title>New Features in Impala for CDH 5.4.x</title>
+
+    <conbody>
+
+      <p>
+        See <xref href="impala_new_features.xml#new_features_220"/> for the most recent set of new Impala features.
+        CDH maintenance releases such as 5.4.1, 5.4.2, and so on are exclusively bug fix releases,
+        therefore there are no new features for the 5.4.x series after 5.4.0.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_22x" />
+
+    </conbody>
+
+  </concept>
+-->
+
+  <concept rev="2.2.0" id="new_features_220">
+
+    <title>New Features in Impala 2.2.x / CDH 5.4.x</title>
+
+    <conbody>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_220" />
+
+      <p>
+        The following are the major new features in Impala 2.2.x. This release, available as part of CDH
+        5.4.x, contains improvements to performance, manageability, security, and SQL syntax.
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            Several improvements to date and time features enable higher interoperability with Hive and other
+            database systems, provide more flexibility for handling time zones, and future-proof the handling of
+            <codeph>TIMESTAMP</codeph> values:
+          </p>
+          <ul>
+            <li>
+              <p>
+                The <codeph>WITH REPLICATION</codeph> clause for the <codeph>CREATE TABLE</codeph> and
+                <codeph>ALTER TABLE</codeph> statements lets you control the replication factor for
+                HDFS caching for a specific table or partition. By default, each cached block is
+                only present on a single host, which can lead to CPU contention if the same host
+                processes each cached block. Increasing the replication factor lets Impala choose
+                different hosts to process different cached blocks, to better distribute the CPU load.
+              </p>
+            </li>
+            <li>
+              <p>
+                Startup flags for the <cmdname>impalad</cmdname> daemon enable a higher level of compatibility with
+                <codeph>TIMESTAMP</codeph> values written by Hive, and more flexibility for working with date and
+                time data using the local time zone instead of UTC. To enable these features, set the
+                <cmdname>impalad</cmdname> startup flags
+                <codeph>-use_local_tz_for_unix_timestamp_conversions=true</codeph> and
+                <codeph>-convert_legacy_hive_parquet_utc_timestamps=true</codeph>.
+              </p>
+
+              <p>
+                The <codeph>-use_local_tz_for_unix_timestamp_conversions</codeph> setting controls how the
+                <codeph>unix_timestamp()</codeph>, <codeph>from_unixtime()</codeph>, and <codeph>now()</codeph>
+                functions handle time zones. By default (when this setting is turned off), Impala considers all
+                <codeph>TIMESTAMP</codeph> values to be in the UTC time zone when converting to or from Unix time
+                values. When this setting is enabled, Impala treats <codeph>TIMESTAMP</codeph> values passed to or
+                returned from these functions to be in the local time zone. When this setting is enabled, take
+                particular care that all hosts in the cluster have the same timezone settings, to avoid
+                inconsistent results depending on which host reads or writes <codeph>TIMESTAMP</codeph> data.
+              </p>
+
+              <p>
+                The <codeph>-convert_legacy_hive_parquet_utc_timestamps</codeph> setting causes Impala to convert
+                <codeph>TIMESTAMP</codeph> values to the local time zone when it reads them from Parquet files
+                written by Hive. This setting only applies to data using the Parquet file format, where Impala can
+                use metadata in the files to reliably determine that the files were written by Hive. If in the
+                future Hive changes the way it writes <codeph>TIMESTAMP</codeph> data in Parquet, Impala will
+                automatically handle that new <codeph>TIMESTAMP</codeph> encoding.
+              </p>
+
+              <p>
+                See <xref href="impala_timestamp.xml#timestamp"/> for details about time zone handling and the
+                configuration options for Impala / Hive compatibility with Parquet format.
+              </p>
+            </li>
+
+            <li>
+              <p conref="../shared/impala_common.xml#common/y2k38" />
+
+              <p>
+                See <xref href="impala_datetime_functions.xml#datetime_functions"/> for the current function
+                signatures.
+              </p>
+            </li>
+          </ul>
+        </li>
+
+        <li>
+          <p>
+            The <codeph>SHOW FILES</codeph> statement lets you view the names and sizes of the files that make up
+            an entire table or a specific partition. See <xref href="impala_show.xml#show_files"/> for details.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Impala can now run queries against Parquet data containing columns with complex or nested types, as
+            long as the query only refers to columns with scalar types.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Performance improvements for queries that include <codeph>IN()</codeph> operators and involve
+            partitioned tables.
+          </p>
+        </li>
+
+        <li>
+<!-- Same text for this item in impala_fixed_issues.xml. Could turn into a conref. -->
+          <p>
+            The new <codeph>-max_log_files</codeph> configuration option specifies how many log files to keep at
+            each severity level. The default value is 10, meaning that Impala preserves the latest 10 log files for
+            each severity level (<codeph>INFO</codeph>, <codeph>WARNING</codeph>, and <codeph>ERROR</codeph>) for
+            each Impala-related daemon (<cmdname>impalad</cmdname>, <cmdname>statestored</cmdname>, and
+            <cmdname>catalogd</cmdname>). Impala checks to see if any old logs need to be removed based on the
+            interval specified in the <codeph>logbufsecs</codeph> setting, every 5 seconds by default. See
+            <xref href="impala_logging.xml#logs_rotate"/> for details.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Redaction of sensitive data from Impala log files. This feature protects details such as credit card
+            numbers or tax IDs from administrators who see the text of SQL statements in the course of monitoring
+            and troubleshooting a Hadoop cluster. See <xref href="impala_logging.xml#redaction"/> for background
+            information for Impala users, and
+            <xref audience="integrated" href="sg_redaction.xml#log_redact"/><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/sg_redaction.html" scope="external" format="html">the CDH 5 Security Guide</xref>
+            for usage details.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Lineage information is available for data created or queried by Impala. This feature lets you track who
+            has accessed data through Impala SQL statements, down to the level of specific columns, and how data
+            has been propagated between tables. See <xref href="impala_lineage.xml#lineage"/> for background
+            information for Impala users,
+            <xref audience="integrated" href="datamgmt_impala_lineage_log.xml"/><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/datamgmt_impala_lineage_log.html" scope="external" format="html"/>
+            for usage details, and
+            <xref audience="integrated" href="cn_iu_lineage.xml" /><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/cn_iu_lineage.html" scope="external" format="html"/>.
+            for how to interpret the lineage
+            information.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Impala tables and partitions can now be located on the Amazon Simple Storage Service (S3) filesystem,
+            for convenience in cases where data is already located in S3 and you prefer to query it in-place.
+            Queries might have lower performance than when the data files reside on HDFS, because Impala uses some
+            HDFS-specific optimizations. Impala can query data in S3, but cannot write to S3. Therefore, statements
+            such as <codeph>INSERT</codeph> and <codeph>LOAD DATA</codeph> are not available when the destination
+            table or partition is in S3. See <xref href="impala_s3.xml#s3"/> for details.
+          </p>
+
+          <note conref="../shared/impala_common.xml#common/s3_caveat" />
+        </li>
+
+        <li>
+        <!-- Only want the link out of the release notes to appear for HTML
+             (N.B. audience="PDF" means hide from PDF), and only in the HTML for the
+             integrated build where the topic is available for link resolution. -->
+          <p>
+            Improved support for HDFS encryption. The <codeph>LOAD DATA</codeph> statement now works when the
+            source directory and destination table are in different encryption zones. <ph audience="integrated"><ph audience="PDF">See
+            <xref href="cdh_sg_component_kms.xml#impala_encryption"/> for details about using HDFS encryption with
+            Impala.</ph></ph>
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Additional arithmetic function <codeph>mod()</codeph>. See
+            <xref href="impala_math_functions.xml#math_functions"/> for details.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Flexibility to interpret <codeph>TIMESTAMP</codeph> values using the UTC time zone (the traditional
+            Impala behavior) or using the local time zone (for compatibility with <codeph>TIMESTAMP</codeph> values
+            produced by Hive).
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Enhanced support for ETL using tools such as Flume. Impala ignores temporary files typically produced
+            by these tools (filenames with suffixes <codeph>.copying</codeph> and <codeph>.tmp</codeph>).
+          </p>
+        </li>
+
+        <li>
+          <p>
+            The CPU requirement for Impala, which had become more restrictive in Impala 2.0.x and 2.1.x, has now
+            been relaxed.
+          </p>
+
+          <p conref="../shared/impala_common.xml#common/cpu_prereq" />
+        </li>
+
+        <li>
+          <p>
+            Enhanced support for <codeph>CHAR</codeph> and <codeph>VARCHAR</codeph> types in the <codeph>COMPUTE
+            STATS</codeph> statement.
+          </p>
+        </li>
+
+        <li rev="CDH-26073">
+          <p>
+            The amount of memory required during setup for <q>spill to disk</q> operations is greatly reduced. This
+            enhancement reduces the chance of a memory-intensive join or aggregation query failing with an
+            out-of-memory error.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Several new conditional functions provide enhanced compatibility when porting code that uses industry
+            extensions. The new functions are: <codeph>isfalse()</codeph>, <codeph>isnotfalse()</codeph>,
+            <codeph>isnottrue()</codeph>, <codeph>istrue()</codeph>, <codeph>nonnullvalue()</codeph>, and
+            <codeph>nullvalue()</codeph>. See <xref href="impala_conditional_functions.xml#conditional_functions"/>
+            for details.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            The Impala debug web UI now can display a visual representation of the query plan. On the
+            <uicontrol>/queries</uicontrol> tab, select <uicontrol>Details</uicontrol> for a particular query. The
+            <uicontrol>Details</uicontrol> page includes a <uicontrol>Plan</uicontrol> tab with a plan diagram that
+            you can zoom in or out (using scroll gestures through mouse wheel or trackpad).
+          </p>
+        </li>
+      </ul>
+
+<!-- End of new feature list for 5.4. -->
+
+    </conbody>
+
+  </concept>
+
+<!-- All 2.1.x subsections go under here -->
+
+  <concept rev="2.1.8" id="new_features_218">
+
+    <title>New Features in Impala 2.1.8 / CDH 5.3.10</title>
+
+    <conbody>
+
+      <p>
+        This point release is exclusively a bug fix release.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.1.7" id="new_features_217">
+
+    <title>New Features in Impala 2.1.7 / CDH 5.3.9</title>
+
+    <conbody>
+
+      <p>
+        This point release is exclusively a bug fix release.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.1.6" id="new_features_216">
+
+    <title>New Features in Impala 2.1.6 / CDH 5.3.8</title>
+
+    <conbody>
+
+      <p>
+        This point release is exclusively a bug fix release.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.1.5" id="new_features_215">
+
+    <title>New Features in Impala 2.1.5 / CDH 5.3.6</title>
+
+    <conbody>
+
+      <p>
+        This point release is exclusively a bug fix release.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.1.4" id="new_features_214">
+
+    <title>New Features in Impala 2.1.4 / CDH 5.3.4</title>
+
+    <conbody>
+
+      <p>
+        No new features. This point release is exclusively a bug fix release.
+        <ph conref="../shared/impala_common.xml#common/impala_214_redux"/>
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.1.3" id="new_features_213">
+
+    <title>New Features in Impala 2.1.3 / CDH 5.3.3</title>
+
+    <conbody>
+
+      <p>
+        No new features. This point release is exclusively a bug fix release.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_213" />
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.1.2" id="new_features_212">
+
+    <title>New Features in Impala 2.1.2 / CDH 5.3.2</title>
+
+    <conbody>
+
+      <p>
+        No new features. This point release is exclusively a bug fix release.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_212" />
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.1.1" id="new_features_211">
+
+    <title>New Features in Impala 2.1.1 / CDH 5.3.1</title>
+
+    <conbody>
+
+      <p>
+        No new features. This point release is exclusively a bug fix release.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.1.0" id="new_features_210">
+
+    <title>New Features in Impala 2.1.0 / CDH 5.3.0</title>
+
+    <conbody>
+
+      <p>
+        This release contains the following enhancements to query performance and system scalability:
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            Impala can now collect statistics for individual partitions in a partitioned table, rather than
+            processing the entire table for each <codeph>COMPUTE STATS</codeph> statement. This feature is known as
+            incremental statistics, and is controlled by the <codeph>COMPUTE INCREMENTAL STATS</codeph> syntax.
+            (You can still use the original <codeph>COMPUTE STATS</codeph> statement for nonpartitioned tables or
+            partitioned tables that are unchanging or whose contents are entirely replaced all at once.) See
+            <xref href="impala_compute_stats.xml#compute_stats"/> and
+            <xref href="impala_perf_stats.xml#perf_stats"/> for details.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Optimization for small queries lets Impala process queries that process very few rows without the
+            unnecessary overhead of parallelizing and generating native code. Reducing this overhead lets Impala
+            clear small queries quickly, keeping YARN resources and admission control slots available for
+            data-intensive queries. The number of rows considered to be a <q>small</q> query is controlled by the
+            <codeph>EXEC_SINGLE_NODE_ROWS_THRESHOLD</codeph> query option. See
+            <xref href="impala_exec_single_node_rows_threshold.xml#exec_single_node_rows_threshold"/> for details.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            An enhancement to the statestore component lets it transmit heartbeat information independently of
+            broadcasting metadata updates. This optimization improves reliability of health checking on large
+            clusters with many tables and partitions.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            The memory requirement for querying gzip-compressed text is reduced. Now Impala decompresses the data
+            as it is read, rather than reading the entire gzipped file and decompressing it in memory.
+          </p>
+        </li>
+      </ul>
+
+    </conbody>
+
+  </concept>
+
+<!-- All 2.0.x subsections go under here -->
+
+  <concept rev="2.0.5" id="new_features_205">
+
+    <title>New Features in Impala 2.0.5 / CDH 5.2.6</title>
+
+    <conbody>
+
+      <p>
+        No new features. This point release is exclusively a bug fix release.
+      </p>
+
+      <note conref=

<TRUNCATED>


[46/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_alter_table.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_alter_table.xml b/docs/topics/impala_alter_table.xml
new file mode 100644
index 0000000..61199cf
--- /dev/null
+++ b/docs/topics/impala_alter_table.xml
@@ -0,0 +1,806 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="alter_table">
+
+  <title>ALTER TABLE Statement</title>
+  <titlealts audience="PDF"><navtitle>ALTER TABLE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="HDFS Caching"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Schemas"/>
+      <data name="Category" value="S3"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">ALTER TABLE statement</indexterm>
+      The <codeph>ALTER TABLE</codeph> statement changes the structure or properties of an existing Impala table.
+    </p>
+    <p>
+      In Impala, this is primarily a logical operation that updates the table metadata in the metastore database that Impala
+      shares with Hive. Most <codeph>ALTER TABLE</codeph> operations do not actually rewrite, move, and so on the actual data
+      files. (The <codeph>RENAME TO</codeph> clause is the one exception; it can cause HDFS files to be moved to different paths.)
+      When you do an <codeph>ALTER TABLE</codeph> operation, you typically need to perform corresponding physical filesystem operations,
+      such as rewriting the data files to include extra fields, or converting them to a different file format.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>ALTER TABLE [<varname>old_db_name</varname>.]<varname>old_table_name</varname> RENAME TO [<varname>new_db_name</varname>.]<varname>new_table_name</varname>
+
+ALTER TABLE <varname>name</varname> ADD COLUMNS (<varname>col_spec</varname>[, <varname>col_spec</varname> ...])
+ALTER TABLE <varname>name</varname> DROP [COLUMN] <varname>column_name</varname>
+ALTER TABLE <varname>name</varname> CHANGE <varname>column_name</varname> <varname>new_name</varname> <varname>new_type</varname>
+ALTER TABLE <varname>name</varname> REPLACE COLUMNS (<varname>col_spec</varname>[, <varname>col_spec</varname> ...])
+
+ALTER TABLE <varname>name</varname> { ADD [IF NOT EXISTS] | DROP [IF EXISTS] } PARTITION (<varname>partition_spec</varname>) <ph rev="2.3.0">[PURGE]</ph>
+<ph rev="2.3.0 IMPALA-1568 CDH-36799">ALTER TABLE <varname>name</varname> RECOVER PARTITIONS</ph>
+
+ALTER TABLE <varname>name</varname> [PARTITION (<varname>partition_spec</varname>)]
+  SET { FILEFORMAT <varname>file_format</varname>
+  | LOCATION '<varname>hdfs_path_of_directory</varname>'
+  | TBLPROPERTIES (<varname>table_properties</varname>)
+  | SERDEPROPERTIES (<varname>serde_properties</varname>) }
+
+<ph rev="2.6.0 IMPALA-3369">ALTER TABLE <varname>name</varname> <varname>colname</varname>
+  ('<varname>statsKey</varname>'='<varname>val</varname>, ...)
+
+statsKey ::= numDVs | numNulls | avgSize | maxSize</ph>
+
+<ph rev="1.4.0">ALTER TABLE <varname>name</varname> [PARTITION (<varname>partition_spec</varname>)] SET { CACHED IN '<varname>pool_name</varname>' <ph rev="2.2.0">[WITH REPLICATION = <varname>integer</varname>]</ph> | UNCACHED }</ph>
+
+<varname>new_name</varname> ::= [<varname>new_database</varname>.]<varname>new_table_name</varname>
+
+<varname>col_spec</varname> ::= <varname>col_name</varname> <varname>type_name</varname>
+
+<varname>partition_spec</varname> ::= <varname>partition_col</varname>=<varname>constant_value</varname>
+
+<varname>table_properties</varname> ::= '<varname>name</varname>'='<varname>value</varname>'[, '<varname>name</varname>'='<varname>value</varname>' ...]
+
+<varname>serde_properties</varname> ::= '<varname>name</varname>'='<varname>value</varname>'[, '<varname>name</varname>'='<varname>value</varname>' ...]
+
+<varname>file_format</varname> ::= { PARQUET | TEXTFILE | RCFILE | SEQUENCEFILE | AVRO }
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+    <p rev="2.3.0">
+      In <keyword keyref="impala23_full"/> and higher, the <codeph>ALTER TABLE</codeph> statement can
+      change the metadata for tables containing complex types (<codeph>ARRAY</codeph>,
+      <codeph>STRUCT</codeph>, and <codeph>MAP</codeph>).
+      For example, you can use an <codeph>ADD COLUMNS</codeph>, <codeph>DROP COLUMN</codeph>, or <codeph>CHANGE</codeph>
+      clause to modify the table layout for complex type columns.
+      Although Impala queries only work for complex type columns in Parquet tables, the complex type support in the
+      <codeph>ALTER TABLE</codeph> statement applies to all file formats.
+      For example, you can use Impala to update metadata for a staging table in a non-Parquet file format where the
+      data is populated by Hive. Or you can use <codeph>ALTER TABLE SET FILEFORMAT</codeph> to change the format
+      of an existing table to Parquet so that Impala can query it. Remember that changing the file format for a table does
+      not convert the data files within the table; you must prepare any Parquet data files containing complex types
+      outside Impala, and bring them into the table using <codeph>LOAD DATA</codeph> or updating the table's
+      <codeph>LOCATION</codeph> property.
+      See <xref href="impala_complex_types.xml#complex_types"/> for details about using complex types.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      Whenever you specify partitions in an <codeph>ALTER TABLE</codeph> statement, through the <codeph>PARTITION
+      (<varname>partition_spec</varname>)</codeph> clause, you must include all the partitioning columns in the
+      specification.
+    </p>
+
+    <p>
+      Most of the <codeph>ALTER TABLE</codeph> operations work the same for internal tables (managed by Impala) as
+      for external tables (with data files located in arbitrary locations). The exception is renaming a table; for
+      an external table, the underlying data directory is not renamed or moved.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/s3_blurb"/>
+
+    <p rev="2.6.0 CDH-39913 IMPALA-1878">
+      You can specify an <codeph>s3a://</codeph> prefix on the <codeph>LOCATION</codeph> attribute of a table or partition
+      to make Impala query data from the Amazon S3 filesystem. In <keyword keyref="impala26_full"/> and higher, Impala automatically
+      handles creating or removing the associated folders when you issue <codeph>ALTER TABLE</codeph> statements
+      with the <codeph>ADD PARTITION</codeph> or <codeph>DROP PARTITION</codeph> clauses.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/s3_ddl"/>
+
+    <p rev="1.4.0">
+      <b>HDFS caching (CACHED IN clause):</b>
+    </p>
+
+    <p rev="1.4.0">
+      If you specify the <codeph>CACHED IN</codeph> clause, any existing or future data files in the table
+      directory or the partition subdirectories are designated to be loaded into memory with the HDFS caching
+      mechanism. See <xref href="impala_perf_hdfs_caching.xml#hdfs_caching"/> for details about using the HDFS
+      caching feature.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/impala_cache_replication_factor"/>
+
+    <p conref="../shared/impala_common.xml#common/sync_ddl_blurb"/>
+
+    <p>
+      The following sections show examples of the use cases for various <codeph>ALTER TABLE</codeph> clauses.
+    </p>
+
+    <p>
+      <b>To rename a table (RENAME TO clause):</b>
+    </p>
+
+<!-- Beefing up the syntax in its original location up to, don't need to repeat it here.
+<codeblock>ALTER TABLE <varname>old_name</varname> RENAME TO <varname>new_name</varname>;</codeblock>
+-->
+
+    <p>
+      The <codeph>RENAME TO</codeph> clause lets you change the name of an existing table, and optionally which
+      database it is located in.
+    </p>
+
+    <p>
+      For internal tables, this operation physically renames the directory within HDFS that contains the data files;
+      the original directory name no longer exists. By qualifying the table names with database names, you can use
+      this technique to move an internal table (and its associated data directory) from one database to another.
+      For example:
+    </p>
+
+<codeblock>create database d1;
+create database d2;
+create database d3;
+use d1;
+create table mobile (x int);
+use d2;
+-- Move table from another database to the current one.
+alter table d1.mobile rename to mobile;
+use d1;
+-- Move table from one database to another.
+alter table d2.mobile rename to d3.mobile;</codeblock>
+
+    <p>
+      For external tables,
+    </p>
+
+    <p>
+      <b>To change the physical location where Impala looks for data files associated with a table or
+      partition:</b>
+    </p>
+
+<codeblock>ALTER TABLE <varname>table_name</varname> [PARTITION (<varname>partition_spec</varname>)] SET LOCATION '<varname>hdfs_path_of_directory</varname>';</codeblock>
+
+    <p>
+      The path you specify is the full HDFS path where the data files reside, or will be created. Impala does not
+      create any additional subdirectory named after the table. Impala does not move any data files to this new
+      location or change any data files that might already exist in that directory.
+    </p>
+
+    <p>
+      To set the location for a single partition, include the <codeph>PARTITION</codeph> clause. Specify all the
+      same partitioning columns for the table, with a constant value for each, to precisely identify the single
+      partition affected by the statement:
+    </p>
+
+<codeblock>create table p1 (s string) partitioned by (month int, day int);
+-- Each ADD PARTITION clause creates a subdirectory in HDFS.
+alter table p1 add partition (month=1, day=1);
+alter table p1 add partition (month=1, day=2);
+alter table p1 add partition (month=2, day=1);
+alter table p1 add partition (month=2, day=2);
+-- Redirect queries, INSERT, and LOAD DATA for one partition
+-- to a specific different directory.
+alter table p1 partition (month=1, day=1) set location '/usr/external_data/new_years_day';
+</codeblock>
+
+    <note conref="../shared/impala_common.xml#common/add_partition_set_location"/>
+
+    <p rev="2.3.0 IMPALA-1568 CDH-36799">
+      <b>To automatically detect new partition directories added through Hive or HDFS operations:</b>
+    </p>
+
+    <p rev="2.3.0 IMPALA-1568 CDH-36799">
+      In <keyword keyref="impala23_full"/> and higher, the <codeph>RECOVER PARTITIONS</codeph> clause scans
+      a partitioned table to detect if any new partition directories were added outside of Impala,
+      such as by Hive <codeph>ALTER TABLE</codeph> statements or by <cmdname>hdfs dfs</cmdname>
+      or <cmdname>hadoop fs</cmdname> commands. The <codeph>RECOVER PARTITIONS</codeph> clause
+      automatically recognizes any data files present in these new directories, the same as
+      the <codeph>REFRESH</codeph> statement does.
+    </p>
+
+    <p rev="2.3.0 IMPALA-1568 CDH-36799">
+      For example, here is a sequence of examples showing how you might create a partitioned table in Impala,
+      create new partitions through Hive, copy data files into the new partitions with the <cmdname>hdfs</cmdname>
+      command, and have Impala recognize the new partitions and new data:
+    </p>
+
+    <p rev="2.3.0 IMPALA-1568 CDH-36799">
+      In Impala, create the table, and a single partition for demonstration purposes:
+    </p>
+
+<codeblock rev="2.3.0 IMPALA-1568 CDH-36799">
+<![CDATA[
+create database recover_partitions;
+use recover_partitions;
+create table t1 (s string) partitioned by (yy int, mm int);
+insert into t1 partition (yy = 2016, mm = 1) values ('Partition exists');
+show files in t1;
++---------------------------------------------------------------------+------+--------------+
+| Path                                                                | Size | Partition    |
++---------------------------------------------------------------------+------+--------------+
+| /user/hive/warehouse/recover_partitions.db/t1/yy=2016/mm=1/data.txt | 17B  | yy=2016/mm=1 |
++---------------------------------------------------------------------+------+--------------+
+quit;
+]]>
+</codeblock>
+
+    <p rev="2.3.0 IMPALA-1568 CDH-36799">
+      In Hive, create some new partitions. In a real use case, you might create the
+      partitions and populate them with data as the final stages of an ETL pipeline.
+    </p>
+
+<codeblock rev="2.3.0 IMPALA-1568 CDH-36799">
+<![CDATA[
+hive> use recover_partitions;
+OK
+hive> alter table t1 add partition (yy = 2016, mm = 2);
+OK
+hive> alter table t1 add partition (yy = 2016, mm = 3);
+OK
+hive> quit;
+]]>
+</codeblock>
+
+    <p rev="2.3.0 IMPALA-1568 CDH-36799">
+      For demonstration purposes, manually copy data (a single row) into these
+      new partitions, using manual HDFS operations:
+    </p>
+
+<codeblock rev="2.3.0 IMPALA-1568 CDH-36799">
+<![CDATA[
+$ hdfs dfs -ls /user/hive/warehouse/recover_partitions.db/t1/yy=2016/
+Found 3 items
+drwxr-xr-x - impala   hive 0 2016-05-09 16:06 /user/hive/warehouse/recover_partitions.db/t1/yy=2016/mm=1
+drwxr-xr-x - jrussell hive 0 2016-05-09 16:14 /user/hive/warehouse/recover_partitions.db/t1/yy=2016/mm=2
+drwxr-xr-x - jrussell hive 0 2016-05-09 16:13 /user/hive/warehouse/recover_partitions.db/t1/yy=2016/mm=3
+
+$ hdfs dfs -cp /user/hive/warehouse/recover_partitions.db/t1/yy=2016/mm=1/data.txt \
+  /user/hive/warehouse/recover_partitions.db/t1/yy=2016/mm=2/data.txt
+$ hdfs dfs -cp /user/hive/warehouse/recover_partitions.db/t1/yy=2016/mm=1/data.txt \
+  /user/hive/warehouse/recover_partitions.db/t1/yy=2016/mm=3/data.txt
+]]>
+</codeblock>
+
+<codeblock rev="2.3.0 IMPALA-1568 CDH-36799">
+<![CDATA[
+hive> select * from t1;
+OK
+Partition exists  2016  1
+Partition exists  2016  2
+Partition exists  2016  3
+hive> quit;
+]]>
+</codeblock>
+
+    <p rev="2.3.0 IMPALA-1568 CDH-36799">
+      In Impala, initially the partitions and data are not visible.
+      Running <codeph>ALTER TABLE</codeph> with the <codeph>RECOVER PARTITIONS</codeph>
+      clause scans the table data directory to find any new partition directories, and
+      the data files inside them:
+    </p>
+
+<codeblock rev="2.3.0 IMPALA-1568 CDH-36799">
+<![CDATA[
+select * from t1;
++------------------+------+----+
+| s                | yy   | mm |
++------------------+------+----+
+| Partition exists | 2016 | 1  |
++------------------+------+----+
+
+alter table t1 recover partitions;
+select * from t1;
++------------------+------+----+
+| s                | yy   | mm |
++------------------+------+----+
+| Partition exists | 2016 | 1  |
+| Partition exists | 2016 | 3  |
+| Partition exists | 2016 | 2  |
++------------------+------+----+
+]]>
+</codeblock>
+
+    <p rev="1.2">
+      <b>To change the key-value pairs of the TBLPROPERTIES and SERDEPROPERTIES fields:</b>
+    </p>
+
+<codeblock>ALTER TABLE <varname>table_name</varname> SET TBLPROPERTIES ('<varname>key1</varname>'='<varname>value1</varname>', '<varname>key2</varname>'='<varname>value2</varname>'[, ...]);
+ALTER TABLE <varname>table_name</varname> SET SERDEPROPERTIES ('<varname>key1</varname>'='<varname>value1</varname>', '<varname>key2</varname>'='<varname>value2</varname>'[, ...]);</codeblock>
+
+    <p>
+      The <codeph>TBLPROPERTIES</codeph> clause is primarily a way to associate arbitrary user-specified data items
+      with a particular table.
+    </p>
+
+    <p>
+      The <codeph>SERDEPROPERTIES</codeph> clause sets up metadata defining how tables are read or written, needed
+      in some cases by Hive but not used extensively by Impala. You would use this clause primarily to change the
+      delimiter in an existing text table or partition, by setting the <codeph>'serialization.format'</codeph> and
+      <codeph>'field.delim'</codeph> property values to the new delimiter character:
+    </p>
+
+<codeblock>-- This table begins life as pipe-separated text format.
+create table change_to_csv (s1 string, s2 string) row format delimited fields terminated by '|';
+-- Then we change it to a CSV table.
+alter table change_to_csv set SERDEPROPERTIES ('serialization.format'=',', 'field.delim'=',');
+insert overwrite change_to_csv values ('stop','go'), ('yes','no');
+!hdfs dfs -cat 'hdfs://<varname>hostname</varname>:8020/<varname>data_directory</varname>/<varname>dbname</varname>.db/change_to_csv/<varname>data_file</varname>';
+stop,go
+yes,no</codeblock>
+
+    <p>
+      Use the <codeph>DESCRIBE FORMATTED</codeph> statement to see the current values of these properties for an
+      existing table. See <xref href="impala_create_table.xml#create_table"/> for more details about these clauses.
+      See <xref href="impala_perf_stats.xml#perf_table_stats_manual"/> for an example of using table properties to
+      fine-tune the performance-related table statistics.
+    </p>
+      
+    <p>
+      <b>To manually set or update table or column statistics:</b>
+    </p>
+
+    <p>
+      Although for most tables the <codeph>COMPUTE STATS</codeph> or <codeph>COMPUTE INCREMENTAL STATS</codeph>
+      statement is all you need to keep table and column statistics up to date for a table,
+      sometimes for a very large table or one that is updated frequently, the length of time to recompute
+      all the statistics might make it impractical to run those statements as often as needed.
+      As a workaround, you can use the <codeph>ALTER TABLE</codeph> statement to set table statistics
+      at the level of the entire table or a single partition, or column statistics at the level of
+      the entire table.
+    </p>
+
+    <p>
+      You can set the <codeph>numrows</codeph> value for table statistics by changing the
+      <codeph>TBLPROPERTIES</codeph> setting for a table or partition.
+      For example:
+<codeblock conref="../shared/impala_common.xml#common/set_numrows_example"/>
+<codeblock conref="../shared/impala_common.xml#common/set_numrows_partitioned_example"/>
+      See <xref href="impala_perf_stats.xml#perf_table_stats_manual"/> for details.
+    </p>
+
+    <p rev="2.6.0 IMPALA-3369">
+      In <keyword keyref="impala26_full"/> and higher, you can use the <codeph>SET COLUMN STATS</codeph> clause
+      to set a specific stats value for a particular column.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/set_column_stats_example"/>
+
+    <p>
+      <b>To reorganize columns for a table:</b>
+    </p>
+
+<codeblock>ALTER TABLE <varname>table_name</varname> ADD COLUMNS (<varname>column_defs</varname>);
+ALTER TABLE <varname>table_name</varname> REPLACE COLUMNS (<varname>column_defs</varname>);
+ALTER TABLE <varname>table_name</varname> CHANGE <varname>column_name</varname> <varname>new_name</varname> <varname>new_type</varname>;
+ALTER TABLE <varname>table_name</varname> DROP <varname>column_name</varname>;</codeblock>
+
+    <p>
+      The <varname>column_spec</varname> is the same as in the <codeph>CREATE TABLE</codeph> statement: the column
+      name, then its data type, then an optional comment. You can add multiple columns at a time. The parentheses
+      are required whether you add a single column or multiple columns. When you replace columns, all the original
+      column definitions are discarded. You might use this technique if you receive a new set of data files with
+      different data types or columns in a different order. (The data files are retained, so if the new columns are
+      incompatible with the old ones, use <codeph>INSERT OVERWRITE</codeph> or <codeph>LOAD DATA OVERWRITE</codeph>
+      to replace all the data before issuing any further queries.)
+    </p>
+
+    <p rev="CDH-37178">
+      For example, here is how you might add columns to an existing table.
+      The first <codeph>ALTER TABLE</codeph> adds two new columns, and the second
+      <codeph>ALTER TABLE</codeph> adds one new column.
+      A single Impala query reads both the old and new data files, containing different numbers of columns.
+      For any columns not present in a particular data file, all the column values are
+      considered to be <codeph>NULL</codeph>.
+    </p>
+
+<codeblock rev="CDH-37178">
+create table t1 (x int);
+insert into t1 values (1), (2);
+
+alter table t1 add columns (s string, t timestamp);
+insert into t1 values (3, 'three', now());
+
+alter table t1 add columns (b boolean);
+insert into t1 values (4, 'four', now(), true);
+
+select * from t1 order by x;
++---+-------+-------------------------------+------+
+| x | s     | t                             | b    |
++---+-------+-------------------------------+------+
+| 1 | NULL  | NULL                          | NULL |
+| 2 | NULL  | NULL                          | NULL |
+| 3 | three | 2016-05-11 11:19:45.054457000 | NULL |
+| 4 | four  | 2016-05-11 11:20:20.260733000 | true |
++---+-------+-------------------------------+------+
+</codeblock>
+
+    <p>
+      You might use the <codeph>CHANGE</codeph> clause to rename a single column, or to treat an existing column as
+      a different type than before, such as to switch between treating a column as <codeph>STRING</codeph> and
+      <codeph>TIMESTAMP</codeph>, or between <codeph>INT</codeph> and <codeph>BIGINT</codeph>. You can only drop a
+      single column at a time; to drop multiple columns, issue multiple <codeph>ALTER TABLE</codeph> statements, or
+      define the new set of columns with a single <codeph>ALTER TABLE ... REPLACE COLUMNS</codeph> statement.
+    </p>
+
+    <p rev="CDH-37178">
+      The following examples show some safe operations to drop or change columns. Dropping the final column
+      in a table lets Impala ignore the data causing any disruption to existing data files. Changing the type
+      of a column works if existing data values can be safely converted to the new type. The type conversion
+      rules depend on the file format of the underlying table. For example, in a text table, the same value
+      can be interpreted as a <codeph>STRING</codeph> or a numeric value, while in a binary format such as
+      Parquet, the rules are stricter and type conversions only work between certain sizes of integers.
+    </p>
+
+<codeblock rev="CDH-37178">
+create table optional_columns (x int, y int, z int, a1 int, a2 int);
+insert into optional_columns values (1,2,3,0,0), (2,3,4,100,100);
+
+-- When the last column in the table is dropped, Impala ignores the
+-- values that are no longer needed. (Dropping A1 but leaving A2
+-- would cause problems, as we will see in a subsequent example.)
+alter table optional_columns drop column a2;
+alter table optional_columns drop column a1;
+
+select * from optional_columns;
++---+---+---+
+| x | y | z |
++---+---+---+
+| 1 | 2 | 3 |
+| 2 | 3 | 4 |
++---+---+---+
+</codeblock>
+
+<codeblock rev="CDH-37178">
+create table int_to_string (s string, x int);
+insert into int_to_string values ('one', 1), ('two', 2);
+
+-- What was an INT column will now be interpreted as STRING.
+-- This technique works for text tables but not other file formats.
+-- The second X represents the new name of the column, which we keep the same.
+alter table int_to_string change x x string;
+
+-- Once the type is changed, we can insert non-integer values into the X column
+-- and treat that column as a string, for example by uppercasing or concatenating.
+insert into int_to_string values ('three', 'trois');
+select s, upper(x) from int_to_string;
++-------+----------+
+| s     | upper(x) |
++-------+----------+
+| one   | 1        |
+| two   | 2        |
+| three | TROIS    |
++-------+----------+
+</codeblock>
+
+    <p rev="CDH-37178">
+      Remember that Impala does not actually do any conversion for the underlying data files as a result of
+      <codeph>ALTER TABLE</codeph> statements. If you use <codeph>ALTER TABLE</codeph> to create a table
+      layout that does not agree with the contents of the underlying files, you must replace the files
+      yourself, such as using <codeph>LOAD DATA</codeph> to load a new set of data files, or
+      <codeph>INSERT OVERWRITE</codeph> to copy from another table and replace the original data.
+    </p>
+
+    <p rev="CDH-37178">
+      The following example shows what happens if you delete the middle column from a Parquet table containing three columns.
+      The underlying data files still contain three columns of data. Because the columns are interpreted based on their positions in
+      the data file instead of the specific column names, a <codeph>SELECT *</codeph> query now reads the first and second
+      columns from the data file, potentially leading to unexpected results or conversion errors.
+      For this reason, if you expect to someday drop a column, declare it as the last column in the table, where its data
+      can be ignored by queries after the column is dropped. Or, re-run your ETL process and create new data files
+      if you drop or change the type of a column in a way that causes problems with existing data files.
+    </p>
+
+<codeblock rev="CDH-37178">
+-- Parquet table showing how dropping a column can produce unexpected results.
+create table p1 (s1 string, s2 string, s3 string) stored as parquet;
+
+insert into p1 values ('one', 'un', 'uno'), ('two', 'deux', 'dos'),
+  ('three', 'trois', 'tres');
+select * from p1;
++-------+-------+------+
+| s1    | s2    | s3   |
++-------+-------+------+
+| one   | un    | uno  |
+| two   | deux  | dos  |
+| three | trois | tres |
++-------+-------+------+
+
+alter table p1 drop column s2;
+-- The S3 column contains unexpected results.
+-- Because S2 and S3 have compatible types, the query reads
+-- values from the dropped S2, because the existing data files
+-- still contain those values as the second column.
+select * from p1;
++-------+-------+
+| s1    | s3    |
++-------+-------+
+| one   | un    |
+| two   | deux  |
+| three | trois |
++-------+-------+
+</codeblock>
+
+<codeblock rev="CDH-37178">
+-- Parquet table showing how dropping a column can produce conversion errors.
+create table p2 (s1 string, x int, s3 string) stored as parquet;
+
+insert into p2 values ('one', 1, 'uno'), ('two', 2, 'dos'), ('three', 3, 'tres');
+select * from p2;
++-------+---+------+
+| s1    | x | s3   |
++-------+---+------+
+| one   | 1 | uno  |
+| two   | 2 | dos  |
+| three | 3 | tres |
++-------+---+------+
+
+alter table p2 drop column x;
+select * from p2;
+WARNINGS: 
+File '<varname>hdfs_filename</varname>' has an incompatible Parquet schema for column 'add_columns.p2.s3'.
+Column type: STRING, Parquet schema:
+optional int32 x [i:1 d:1 r:0]
+
+File '<varname>hdfs_filename</varname>' has an incompatible Parquet schema for column 'add_columns.p2.s3'.
+Column type: STRING, Parquet schema:
+optional int32 x [i:1 d:1 r:0]
+</codeblock>
+
+    <p rev="IMPALA-3092">
+      In <keyword keyref="impala26_full"/> and higher, if an Avro table is created without column definitions in the
+      <codeph>CREATE TABLE</codeph> statement, and columns are later
+      added through <codeph>ALTER TABLE</codeph>, the resulting
+      table is now queryable. Missing values from the newly added
+      columns now default to <codeph>NULL</codeph>.
+    </p>
+
+    <p>
+      <b>To change the file format that Impala expects data to be in, for a table or partition:</b>
+    </p>
+
+    <p>
+      Use an <codeph>ALTER TABLE ... SET FILEFORMAT</codeph> clause. You can include an optional <codeph>PARTITION
+      (<varname>col1</varname>=<varname>val1</varname>, <varname>col2</varname>=<varname>val2</varname>,
+      ...</codeph> clause so that the file format is changed for a specific partition rather than the entire table.
+    </p>
+
+    <p>
+      Because this operation only changes the table metadata, you must do any conversion of existing data using
+      regular Hadoop techniques outside of Impala. Any new data created by the Impala <codeph>INSERT</codeph>
+      statement will be in the new format. You cannot specify the delimiter for Text files; the data files must be
+      comma-delimited.
+<!-- Although Impala can read Avro tables
+        created through Hive, you cannot specify the Avro file format in an Impala
+        <codeph>ALTER TABLE</codeph> statement. -->
+    </p>
+
+    <p>
+      To set the file format for a single partition, include the <codeph>PARTITION</codeph> clause. Specify all the
+      same partitioning columns for the table, with a constant value for each, to precisely identify the single
+      partition affected by the statement:
+    </p>
+
+<codeblock>create table p1 (s string) partitioned by (month int, day int);
+-- Each ADD PARTITION clause creates a subdirectory in HDFS.
+alter table p1 add partition (month=1, day=1);
+alter table p1 add partition (month=1, day=2);
+alter table p1 add partition (month=2, day=1);
+alter table p1 add partition (month=2, day=2);
+-- Queries and INSERT statements will read and write files
+-- in this format for this specific partition.
+alter table p1 partition (month=2, day=2) set fileformat parquet;
+</codeblock>
+
+    <p>
+      <b>To add or drop partitions for a table</b>, the table must already be partitioned (that is, created with a
+      <codeph>PARTITIONED BY</codeph> clause). The partition is a physical directory in HDFS, with a name that
+      encodes a particular column value (the <b>partition key</b>). The Impala <codeph>INSERT</codeph> statement
+      already creates the partition if necessary, so the <codeph>ALTER TABLE ... ADD PARTITION</codeph> is
+      primarily useful for importing data by moving or copying existing data files into the HDFS directory
+      corresponding to a partition. (You can use the <codeph>LOAD DATA</codeph> statement to move files into the
+      partition directory, or <codeph>ALTER TABLE ... PARTITION (...) SET LOCATION</codeph> to point a partition at
+      a directory that already contains data files.
+    </p>
+
+    <p>
+      The <codeph>DROP PARTITION</codeph> clause is used to remove the HDFS directory and associated data files for
+      a particular set of partition key values; for example, if you always analyze the last 3 months worth of data,
+      at the beginning of each month you might drop the oldest partition that is no longer needed. Removing
+      partitions reduces the amount of metadata associated with the table and the complexity of calculating the
+      optimal query plan, which can simplify and speed up queries on partitioned tables, particularly join queries.
+      Here is an example showing the <codeph>ADD PARTITION</codeph> and <codeph>DROP PARTITION</codeph> clauses.
+    </p>
+
+    <p>
+      To avoid errors while adding or dropping partitions whose existence is not certain,
+      add the optional <codeph>IF [NOT] EXISTS</codeph> clause between the <codeph>ADD</codeph> or
+      <codeph>DROP</codeph> keyword and the <codeph>PARTITION</codeph> keyword. That is, the entire
+      clause becomes <codeph>ADD IF NOT EXISTS PARTITION</codeph> or <codeph>DROP IF EXISTS PARTITION</codeph>.
+      The following example shows how partitions can be created automatically through <codeph>INSERT</codeph>
+      statements, or manually through <codeph>ALTER TABLE</codeph> statements. The <codeph>IF [NOT] EXISTS</codeph>
+      clauses let the <codeph>ALTER TABLE</codeph> statements succeed even if a new requested partition already
+      exists, or a partition to be dropped does not exist.
+    </p>
+
+<p>
+Inserting 2 year values creates 2 partitions:
+</p>
+
+<codeblock>
+create table partition_t (s string) partitioned by (y int);
+insert into partition_t (s,y) values ('two thousand',2000), ('nineteen ninety',1990);
+show partitions partition_t;
++-------+-------+--------+------+--------------+-------------------+--------+-------------------+
+| y     | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats |
++-------+-------+--------+------+--------------+-------------------+--------+-------+
+| 1990  | -1    | 1      | 16B  | NOT CACHED   | NOT CACHED        | TEXT   | false |
+| 2000  | -1    | 1      | 13B  | NOT CACHED   | NOT CACHED        | TEXT   | false |
+| Total | -1    | 2      | 29B  | 0B           |                   |        |       |
++-------+-------+--------+------+--------------+-------------------+--------+-------+
+</codeblock>
+
+<p>
+Without the <codeph>IF NOT EXISTS</codeph> clause, an attempt to add a new partition might fail:
+</p>
+
+<codeblock>
+alter table partition_t add partition (y=2000);
+ERROR: AnalysisException: Partition spec already exists: (y=2000).
+</codeblock>
+
+<p>
+The <codeph>IF NOT EXISTS</codeph> clause makes the statement succeed whether or not there was already a
+partition with the specified key value:
+</p>
+
+<codeblock>
+alter table partition_t add if not exists partition (y=2000);
+alter table partition_t add if not exists partition (y=2010);
+show partitions partition_t;
++-------+-------+--------+------+--------------+-------------------+--------+-------------------+
+| y     | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats |
++-------+-------+--------+------+--------------+-------------------+--------+-------+
+| 1990  | -1    | 1      | 16B  | NOT CACHED   | NOT CACHED        | TEXT   | false |
+| 2000  | -1    | 1      | 13B  | NOT CACHED   | NOT CACHED        | TEXT   | false |
+| 2010  | -1    | 0      | 0B   | NOT CACHED   | NOT CACHED        | TEXT   | false |
+| Total | -1    | 2      | 29B  | 0B           |                   |        |       |
++-------+-------+--------+------+--------------+-------------------+--------+-------+
+</codeblock>
+
+<p>
+Likewise, the <codeph>IF EXISTS</codeph> clause lets <codeph>DROP PARTITION</codeph> succeed whether or not the partition is already
+in the table:
+</p>
+
+<codeblock>
+alter table partition_t drop if exists partition (y=2000);
+alter table partition_t drop if exists partition (y=1950);
+show partitions partition_t;
++-------+-------+--------+------+--------------+-------------------+--------+-------------------+
+| y     | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats |
++-------+-------+--------+------+--------------+-------------------+--------+-------+
+| 1990  | -1    | 1      | 16B  | NOT CACHED   | NOT CACHED        | TEXT   | false |
+| 2010  | -1    | 0      | 0B   | NOT CACHED   | NOT CACHED        | TEXT   | false |
+| Total | -1    | 1      | 16B  | 0B           |                   |        |       |
++-------+-------+--------+------+--------------+-------------------+--------+-------+
+</codeblock>
+
+    <p rev="2.3.0"> The optional <codeph>PURGE</codeph> keyword, available in
+      <keyword keyref="impala23_full"/> and higher, is used with the <codeph>DROP
+        PARTITION</codeph> clause to remove associated HDFS data files
+      immediately rather than going through the HDFS trashcan mechanism. Use
+      this keyword when dropping a partition if it is crucial to remove the data
+      as quickly as possible to free up space, or if there is a problem with the
+      trashcan, such as the trash cannot being configured or being in a
+      different HDFS encryption zone than the data files. </p>
+
+    <!--
+        To do: Make example more general by partitioning by year/month/day.
+        Then could show inserting into fixed year, variable month and day;
+        dropping particular year/month/day partition.
+    -->
+
+<codeblock>-- Create an empty table and define the partitioning scheme.
+create table part_t (x int) partitioned by (month int);
+-- Create an empty partition into which you could copy data files from some other source.
+alter table part_t add partition (month=1);
+-- After changing the underlying data, issue a REFRESH statement to make the data visible in Impala.
+refresh part_t;
+-- Later, do the same for the next month.
+alter table part_t add partition (month=2);
+
+-- Now you no longer need the older data.
+alter table part_t drop partition (month=1);
+-- If the table was partitioned by month and year, you would issue a statement like:
+-- alter table part_t drop partition (year=2003,month=1);
+-- which would require 12 ALTER TABLE statements to remove a year's worth of data.
+
+-- If the data files for subsequent months were in a different file format,
+-- you could set a different file format for the new partition as you create it.
+alter table part_t add partition (month=3) set fileformat=parquet;
+</codeblock>
+
+    <p>
+      The value specified for a partition key can be an arbitrary constant expression, without any references to
+      columns. For example:
+    </p>
+
+<codeblock>alter table time_data add partition (month=concat('Decem','ber'));
+alter table sales_data add partition (zipcode = cast(9021 * 10 as string));</codeblock>
+
+    <note>
+      <p>
+        An alternative way to reorganize a table and its associated data files is to use <codeph>CREATE
+        TABLE</codeph> to create a variation of the original table, then use <codeph>INSERT</codeph> to copy the
+        transformed or reordered data to the new table. The advantage of <codeph>ALTER TABLE</codeph> is that it
+        avoids making a duplicate copy of the data files, allowing you to reorganize huge volumes of data in a
+        space-efficient way using familiar Hadoop techniques.
+      </p>
+    </note>
+
+    <p>
+      <b>To switch a table between internal and external:</b>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/switch_internal_external_table"/>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+    <p rev="CDH-19187">
+      Most <codeph>ALTER TABLE</codeph> clauses do not actually
+      read or write any HDFS files, and so do not depend on
+      specific HDFS permissions. For example, the <codeph>SET FILEFORMAT</codeph>
+      clause does not actually check the file format existing data files or
+      convert them to the new format, and the <codeph>SET LOCATION</codeph> clause
+      does not require any special permissions on the new location.
+      (Any permission-related failures would come later, when you
+      actually query or insert into the table.)
+    </p>
+<!-- Haven't rigorously tested all the assertions in the following paragraph. -->
+<!-- Most testing so far has been around RENAME TO clause. -->
+    <p>
+      In general, <codeph>ALTER TABLE</codeph> clauses that do touch
+      HDFS files and directories require the same HDFS permissions
+      as corresponding <codeph>CREATE</codeph>, <codeph>INSERT</codeph>,
+      or <codeph>SELECT</codeph> statements.
+      The permissions allow
+      the user ID that the <cmdname>impalad</cmdname> daemon runs under,
+      typically the <codeph>impala</codeph> user, to read or write
+      files or directories, or (in the case of the execute bit) descend into a directory.
+      The <codeph>RENAME TO</codeph> clause requires read, write, and execute permission in the
+      source and destination database directories and in the table data directory,
+      and read and write permission for the data files within the table.
+      The <codeph>ADD PARTITION</codeph> and <codeph>DROP PARTITION</codeph> clauses
+      require write and execute permissions for the associated partition directory.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_tables.xml#tables"/>,
+      <xref href="impala_create_table.xml#create_table"/>, <xref href="impala_drop_table.xml#drop_table"/>,
+      <xref href="impala_partitioning.xml#partitioning"/>, <xref href="impala_tables.xml#internal_tables"/>,
+      <xref href="impala_tables.xml#external_tables"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_alter_view.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_alter_view.xml b/docs/topics/impala_alter_view.xml
new file mode 100644
index 0000000..baf539d
--- /dev/null
+++ b/docs/topics/impala_alter_view.xml
@@ -0,0 +1,86 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.1" id="alter_view">
+
+  <title>ALTER VIEW Statement</title>
+  <titlealts audience="PDF"><navtitle>ALTER VIEW</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Schemas"/>
+      <data name="Category" value="Views"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">ALTER VIEW statement</indexterm>
+      Changes the characteristics of a view. The syntax has two forms:
+    </p>
+
+    <ul>
+      <li>
+        The <codeph>AS</codeph> clause associates the view with a different query.
+      </li>
+      <li>
+        The <codeph>RENAME TO</codeph> clause changes the name of the view, moves the view to
+        a different database, or both.
+      </li>
+    </ul>
+
+    <p>
+      Because a view is purely a logical construct (an alias for a query) with no physical data behind it,
+      <codeph>ALTER VIEW</codeph> only involves changes to metadata in the metastore database, not any data files
+      in HDFS.
+    </p>
+
+<!-- View _permissions_ don't rely on underlying table. -->
+
+<!-- Could use views to grant access only to certain columns. -->
+
+<!-- Treated like a table for authorization. -->
+
+<!-- ALTER VIEW that queries another view - possibly a runtime error. -->
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>ALTER VIEW [<varname>database_name</varname>.]<varname>view_name</varname> AS <varname>select_statement</varname>
+ALTER VIEW [<varname>database_name</varname>.]<varname>view_name</varname> RENAME TO [<varname>database_name</varname>.]<varname>view_name</varname></codeblock>
+
+    <p conref="../shared/impala_common.xml#common/ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/sync_ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/security_blurb"/>
+    <p conref="../shared/impala_common.xml#common/redaction_yes"/>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>create table t1 (x int, y int, s string);
+create table t2 like t1;
+create view v1 as select * from t1;
+alter view v1 as select * from t2;
+alter view v1 as select x, upper(s) s from t2;</codeblock>
+
+<!-- Repeat the same blurb + example to see the definition of a view, as in CREATE VIEW. -->
+
+    <p conref="../shared/impala_common.xml#common/describe_formatted_view"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_views.xml#views"/>, <xref href="impala_create_view.xml#create_view"/>,
+      <xref href="impala_drop_view.xml#drop_view"/>
+    </p>
+  </conbody>
+</concept>



[35/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_ddl.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_ddl.xml b/docs/topics/impala_ddl.xml
new file mode 100644
index 0000000..8e6a3bd
--- /dev/null
+++ b/docs/topics/impala_ddl.xml
@@ -0,0 +1,150 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="ddl">
+
+  <title>DDL Statements</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Schemas"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Databases"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      DDL refers to <q>Data Definition Language</q>, a subset of SQL statements that change the structure of the
+      database schema in some way, typically by creating, deleting, or modifying schema objects such as databases,
+      tables, and views. Most Impala DDL statements start with the keywords <codeph>CREATE</codeph>,
+      <codeph>DROP</codeph>, or <codeph>ALTER</codeph>.
+    </p>
+
+    <p>
+      The Impala DDL statements are:
+    </p>
+
+    <ul>
+      <li>
+        <xref href="impala_alter_table.xml#alter_table"/>
+      </li>
+
+      <li>
+        <xref href="impala_alter_view.xml#alter_view"/>
+      </li>
+
+      <li>
+        <xref href="impala_compute_stats.xml#compute_stats"/>
+      </li>
+
+      <li>
+        <xref href="impala_create_database.xml#create_database"/>
+      </li>
+
+      <li>
+        <xref href="impala_create_function.xml#create_function"/>
+      </li>
+
+      <li rev="2.0.0">
+        <xref href="impala_create_role.xml#create_role"/>
+      </li>
+
+      <li>
+        <xref href="impala_create_table.xml#create_table"/>
+      </li>
+
+      <li>
+        <xref href="impala_create_view.xml#create_view"/>
+      </li>
+
+      <li>
+        <xref href="impala_drop_database.xml#drop_database"/>
+      </li>
+
+      <li>
+        <xref href="impala_drop_function.xml#drop_function"/>
+      </li>
+
+      <li rev="2.0.0">
+        <xref href="impala_drop_role.xml#drop_role"/>
+      </li>
+
+      <li>
+        <xref href="impala_drop_table.xml#drop_table"/>
+      </li>
+
+      <li>
+        <xref href="impala_drop_view.xml#drop_view"/>
+      </li>
+
+      <li rev="2.0.0">
+        <xref href="impala_grant.xml#grant"/>
+      </li>
+
+      <li rev="2.0.0">
+        <xref href="impala_revoke.xml#revoke"/>
+      </li>
+    </ul>
+
+    <p>
+      After Impala executes a DDL command, information about available tables, columns, views, partitions, and so
+      on is automatically synchronized between all the Impala nodes in a cluster. (Prior to Impala 1.2, you had to
+      issue a <codeph>REFRESH</codeph> or <codeph>INVALIDATE METADATA</codeph> statement manually on the other
+      nodes to make them aware of the changes.)
+    </p>
+
+    <p>
+      If the timing of metadata updates is significant, for example if you use round-robin scheduling where each
+      query could be issued through a different Impala node, you can enable the
+      <xref href="impala_sync_ddl.xml#sync_ddl">SYNC_DDL</xref> query option to make the DDL statement wait until
+      all nodes have been notified about the metadata changes.
+    </p>
+
+    <p rev="2.2.0">
+      See <xref href="impala_s3.xml#s3"/> for details about how Impala DDL statements interact with
+      tables and partitions stored in the Amazon S3 filesystem.
+    </p>
+
+    <p>
+      Although the <codeph>INSERT</codeph> statement is officially classified as a DML (data manipulation language)
+      statement, it also involves metadata changes that must be broadcast to all Impala nodes, and so is also
+      affected by the <codeph>SYNC_DDL</codeph> query option.
+    </p>
+
+    <p>
+      Because the <codeph>SYNC_DDL</codeph> query option makes each DDL operation take longer than normal, you
+      might only enable it before the last DDL operation in a sequence. For example, if you are running a script
+      that issues multiple of DDL operations to set up an entire new schema, add several new partitions, and so on,
+      you might minimize the performance overhead by enabling the query option only before the last
+      <codeph>CREATE</codeph>, <codeph>DROP</codeph>, <codeph>ALTER</codeph>, or <codeph>INSERT</codeph> statement.
+      The script only finishes when all the relevant metadata changes are recognized by all the Impala nodes, so
+      you could connect to any node and issue queries through it.
+    </p>
+
+    <p>
+      The classification of DDL, DML, and other statements is not necessarily the same between Impala and Hive.
+      Impala organizes these statements in a way intended to be familiar to people familiar with relational
+      databases or data warehouse products. Statements that modify the metastore database, such as <codeph>COMPUTE
+      STATS</codeph>, are classified as DDL. Statements that only query the metastore database, such as
+      <codeph>SHOW</codeph> or <codeph>DESCRIBE</codeph>, are put into a separate category of utility statements.
+    </p>
+
+    <note>
+      The query types shown in the Impala debug web user interface might not match exactly the categories listed
+      here. For example, currently the <codeph>USE</codeph> statement is shown as DDL in the debug web UI. The
+      query types shown in the debug web UI are subject to change, for improved consistency.
+    </note>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      The other major classifications of SQL statements are data manipulation language (see
+      <xref href="impala_dml.xml#dml"/>) and queries (see <xref href="impala_select.xml#select"/>).
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_debug_action.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_debug_action.xml b/docs/topics/impala_debug_action.xml
new file mode 100644
index 0000000..977cbe0
--- /dev/null
+++ b/docs/topics/impala_debug_action.xml
@@ -0,0 +1,33 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="debug_action">
+
+  <title>DEBUG_ACTION Query Option</title>
+  <titlealts audience="PDF"><navtitle>DEBUG_ACTION</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Troubleshooting"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">DEBUG_ACTION query option</indexterm>
+      Introduces artificial problem conditions within queries. For internal Cloudera debugging and troubleshooting.
+    </p>
+
+    <p>
+      <b>Type:</b> <codeph>STRING</codeph>
+    </p>
+
+    <p>
+      <b>Default:</b> empty string
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_decimal.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_decimal.xml b/docs/topics/impala_decimal.xml
new file mode 100644
index 0000000..58958ab
--- /dev/null
+++ b/docs/topics/impala_decimal.xml
@@ -0,0 +1,817 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.4.0" id="decimal">
+
+  <title>DECIMAL Data Type (<keyword keyref="impala14"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>DECIMAL</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      A numeric data type with fixed scale and precision, used in <codeph>CREATE TABLE</codeph> and <codeph>ALTER
+      TABLE</codeph> statements. Suitable for financial and other arithmetic calculations where the imprecise
+      representation and rounding behavior of <codeph>FLOAT</codeph> and <codeph>DOUBLE</codeph> make those types
+      impractical.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p>
+      In the column definition of a <codeph>CREATE TABLE</codeph> statement:
+    </p>
+
+<codeblock><varname>column_name</varname> DECIMAL[(<varname>precision</varname>[,<varname>scale</varname>])]</codeblock>
+
+    <p>
+      <codeph>DECIMAL</codeph> with no precision or scale values is equivalent to <codeph>DECIMAL(9,0)</codeph>.
+    </p>
+
+    <p>
+      <b>Precision and Scale:</b>
+    </p>
+
+    <p>
+      <varname>precision</varname> represents the total number of digits that can be represented by the column,
+      regardless of the location of the decimal point. This value must be between 1 and 38. For example,
+      representing integer values up to 9999, and floating-point values up to 99.99, both require a precision of 4.
+      You can also represent corresponding negative values, without any change in the precision. For example, the
+      range -9999 to 9999 still only requires a precision of 4.
+    </p>
+
+    <p>
+      <varname>scale</varname> represents the number of fractional digits. This value must be less than or equal to
+      <varname>precision</varname>. A scale of 0 produces integral values, with no fractional part. If precision
+      and scale are equal, all the digits come after the decimal point, making all the values between 0 and
+      0.999... or 0 and -0.999...
+    </p>
+
+    <p>
+      When <varname>precision</varname> and <varname>scale</varname> are omitted, a <codeph>DECIMAL</codeph> value
+      is treated as <codeph>DECIMAL(9,0)</codeph>, that is, an integer value ranging from
+      <codeph>-999,999,999</codeph> to <codeph>999,999,999</codeph>. This is the largest <codeph>DECIMAL</codeph>
+      value that can still be represented in 4 bytes. If precision is specified but scale is omitted, Impala uses a
+      value of zero for the scale.
+    </p>
+
+    <p>
+      Both <varname>precision</varname> and <varname>scale</varname> must be specified as integer literals, not any
+      other kind of constant expressions.
+    </p>
+
+    <p>
+      To check the precision or scale for arbitrary values, you can call the
+      <xref href="impala_math_functions.xml#math_functions"><codeph>precision()</codeph> and
+      <codeph>scale()</codeph> built-in functions</xref>. For example, you might use these values to figure out how
+      many characters are required for various fields in a report, or to understand the rounding characteristics of
+      a formula as applied to a particular <codeph>DECIMAL</codeph> column.
+    </p>
+
+    <p>
+      <b>Range:</b>
+    </p>
+
+    <p>
+      The maximum precision value is 38. Thus, the largest integral value is represented by
+      <codeph>DECIMAL(38,0)</codeph> (999... with 9 repeated 38 times). The most precise fractional value (between
+      0 and 1, or 0 and -1) is represented by <codeph>DECIMAL(38,38)</codeph>, with 38 digits to the right of the
+      decimal point. The value closest to 0 would be .0000...1 (37 zeros and the final 1). The value closest to 1
+      would be .999... (9 repeated 38 times).
+    </p>
+
+    <p>
+      For a given precision and scale, the range of <codeph>DECIMAL</codeph> values is the same in the positive and
+      negative directions. For example, <codeph>DECIMAL(4,2)</codeph> can represent from -99.99 to 99.99. This is
+      different from other integral numeric types where the positive and negative bounds differ slightly.
+    </p>
+
+    <p>
+      When you use <codeph>DECIMAL</codeph> values in arithmetic expressions, the precision and scale of the result
+      value are determined as follows:
+    </p>
+
+    <ul>
+      <li>
+        <p>
+          For addition and subtraction, the precision and scale are based on the maximum possible result, that is,
+          if all the digits of the input values were 9s and the absolute values were added together.
+        </p>
+<!-- Seems like buggy output from this first query, so hiding the example for the time being. -->
+<codeblock audience="Cloudera"><![CDATA[[localhost:21000] > select 50000.5 + 12.444, precision(50000.5 + 12.444), scale(50000.5 + 12.444);
++------------------+-----------------------------+-------------------------+
+| 50000.5 + 12.444 | precision(50000.5 + 12.444) | scale(50000.5 + 12.444) |
++------------------+-----------------------------+-------------------------+
+| 50012.944        | 9                           | 3                       |
++------------------+-----------------------------+-------------------------+
+[localhost:21000] > select 99999.9 + 99.999, precision(99999.9 + 99.999), scale(99999.9 + 99.999);
++------------------+-----------------------------+-------------------------+
+| 99999.9 + 99.999 | precision(99999.9 + 99.999) | scale(99999.9 + 99.999) |
++------------------+-----------------------------+-------------------------+
+| 100099.899       | 9                           | 3                       |
++------------------+-----------------------------+-------------------------+
+]]>
+</codeblock>
+      </li>
+
+      <li>
+        <p>
+          For multiplication, the precision is the sum of the precisions of the input values. The scale is the sum
+          of the scales of the input values.
+        </p>
+      </li>
+
+<!-- Need to add some specifics to discussion of division. Details here: http://blogs.msdn.com/b/sqlprogrammability/archive/2006/03/29/564110.aspx -->
+
+      <li>
+        <p>
+          For division, Impala sets the precision and scale to values large enough to represent the whole and
+          fractional parts of the result.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          For <codeph>UNION</codeph>, the scale is the larger of the scales of the input values, and the precision
+          is increased if necessary to accommodate any additional fractional digits. If the same input value has
+          the largest precision and the largest scale, the result value has the same precision and scale. If one
+          value has a larger precision but smaller scale, the scale of the result value is increased. For example,
+          <codeph>DECIMAL(20,2) UNION DECIMAL(8,6)</codeph> produces a result of type
+          <codeph>DECIMAL(24,6)</codeph>. The extra 4 fractional digits of scale (6-2) are accommodated by
+          extending the precision by the same amount (20+4).
+        </p>
+      </li>
+
+      <li>
+        <p>
+          To doublecheck, you can always call the <codeph>PRECISION()</codeph> and <codeph>SCALE()</codeph>
+          functions on the results of an arithmetic expression to see the relevant values, or use a <codeph>CREATE
+          TABLE AS SELECT</codeph> statement to define a column based on the return type of the expression.
+        </p>
+      </li>
+    </ul>
+
+    <p conref="../shared/impala_common.xml#common/compatibility_blurb"/>
+
+    <ul>
+      <li>
+        Using the <codeph>DECIMAL</codeph> type is only supported under <keyword keyref="impala14_full"/> and higher.
+      </li>
+
+      <li>
+        Use the <codeph>DECIMAL</codeph> data type in Impala for applications where you used the
+        <codeph>NUMBER</codeph> data type in Oracle. The Impala <codeph>DECIMAL</codeph> type does not support the
+        Oracle idioms of <codeph>*</codeph> for scale or negative values for precision.
+      </li>
+    </ul>
+
+    <p>
+      <b>Conversions and casting:</b>
+    </p>
+
+    <p>
+      <ph conref="../shared/impala_common.xml#common/cast_int_to_timestamp"/>
+    </p>
+
+    <p>
+      Impala automatically converts between <codeph>DECIMAL</codeph> and other numeric types where possible. A
+      <codeph>DECIMAL</codeph> with zero scale is converted to or from the smallest appropriate integral type. A
+      <codeph>DECIMAL</codeph> with a fractional part is automatically converted to or from the smallest
+      appropriate floating-point type. If the destination type does not have sufficient precision or scale to hold
+      all possible values of the source type, Impala raises an error and does not convert the value.
+    </p>
+
+    <p>
+      For example, these statements show how expressions of <codeph>DECIMAL</codeph> and other types are reconciled
+      to the same type in the context of <codeph>UNION</codeph> queries and <codeph>INSERT</codeph> statements:
+    </p>
+
+<codeblock><![CDATA[[localhost:21000] > select cast(1 as int) as x union select cast(1.5 as decimal(9,4)) as x;
++----------------+
+| x              |
++----------------+
+| 1.5000         |
+| 1.0000         |
++----------------+
+[localhost:21000] > create table int_vs_decimal as select cast(1 as int) as x union select cast(1.5 as decimal(9,4)) as x;
++-------------------+
+| summary           |
++-------------------+
+| Inserted 2 row(s) |
++-------------------+
+[localhost:21000] > desc int_vs_decimal;
++------+---------------+---------+
+| name | type          | comment |
++------+---------------+---------+
+| x    | decimal(14,4) |         |
++------+---------------+---------+
+]]>
+</codeblock>
+
+    <p>
+      To avoid potential conversion errors, you can use <codeph>CAST()</codeph> to convert <codeph>DECIMAL</codeph>
+      values to <codeph>FLOAT</codeph>, <codeph>TINYINT</codeph>, <codeph>SMALLINT</codeph>, <codeph>INT</codeph>,
+      <codeph>BIGINT</codeph>, <codeph>STRING</codeph>, <codeph>TIMESTAMP</codeph>, or <codeph>BOOLEAN</codeph>.
+      You can use exponential notation in <codeph>DECIMAL</codeph> literals or when casting from
+      <codeph>STRING</codeph>, for example <codeph>1.0e6</codeph> to represent one million.
+    </p>
+
+    <p>
+      If you cast a value with more fractional digits than the scale of the destination type, any extra fractional
+      digits are truncated (not rounded). Casting a value to a target type with not enough precision produces a
+      result of <codeph>NULL</codeph> and displays a runtime warning.
+    </p>
+
+<codeblock><![CDATA[[localhost:21000] > select cast(1.239 as decimal(3,2));
++-----------------------------+
+| cast(1.239 as decimal(3,2)) |
++-----------------------------+
+| 1.23                        |
++-----------------------------+
+[localhost:21000] > select cast(1234 as decimal(3));
++----------------------------+
+| cast(1234 as decimal(3,0)) |
++----------------------------+
+| NULL                       |
++----------------------------+
+WARNINGS: Expression overflowed, returning NULL
+]]>
+</codeblock>
+
+    <p>
+      When you specify integer literals, for example in <codeph>INSERT ... VALUES</codeph> statements or arithmetic
+      expressions, those numbers are interpreted as the smallest applicable integer type. You must use
+      <codeph>CAST()</codeph> calls for some combinations of integer literals and <codeph>DECIMAL</codeph>
+      precision. For example, <codeph>INT</codeph> has a maximum value that is 10 digits long,
+      <codeph>TINYINT</codeph> has a maximum value that is 3 digits long, and so on. If you specify a value such as
+      123456 to go into a <codeph>DECIMAL</codeph> column, Impala checks if the column has enough precision to
+      represent the largest value of that integer type, and raises an error if not. Therefore, use an expression
+      like <codeph>CAST(123456 TO DECIMAL(9,0))</codeph> for <codeph>DECIMAL</codeph> columns with precision 9 or
+      less, <codeph>CAST(50 TO DECIMAL(2,0))</codeph> for <codeph>DECIMAL</codeph> columns with precision 2 or
+      less, and so on. For <codeph>DECIMAL</codeph> columns with precision 10 or greater, Impala automatically
+      interprets the value as the correct <codeph>DECIMAL</codeph> type; however, because
+      <codeph>DECIMAL(10)</codeph> requires 8 bytes of storage while <codeph>DECIMAL(9)</codeph> requires only 4
+      bytes, only use precision of 10 or higher when actually needed.
+    </p>
+
+<codeblock><![CDATA[[localhost:21000] > create table decimals_9_0 (x decimal);
+[localhost:21000] > insert into decimals_9_0 values (1), (2), (4), (8), (16), (1024), (32768), (65536), (1000000);
+ERROR: AnalysisException: Possible loss of precision for target table 'decimal_testing.decimals_9_0'.
+Expression '1' (type: INT) would need to be cast to DECIMAL(9,0) for column 'x'
+[localhost:21000] > insert into decimals_9_0 values (cast(1 as decimal)), (cast(2 as decimal)), (cast(4 as decimal)), (cast(8 as decimal)), (cast(16 as decimal)), (cast(1024 as decimal)), (cast(32768 as decimal)), (cast(65536 as decimal)), (cast(1000000 as decimal));
+
+[localhost:21000] > create table decimals_10_0 (x decimal(10,0));
+[localhost:21000] > insert into decimals_10_0 values (1), (2), (4), (8), (16), (1024), (32768), (65536), (1000000);
+]]>
+</codeblock>
+
+    <p>
+      Be aware that in memory and for binary file formats such as Parquet or Avro, <codeph>DECIMAL(10)</codeph> or
+      higher consumes 8 bytes while <codeph>DECIMAL(9)</codeph> (the default for <codeph>DECIMAL</codeph>) or lower
+      consumes 4 bytes. Therefore, to conserve space in large tables, use the smallest-precision
+      <codeph>DECIMAL</codeph> type that is appropriate and <codeph>CAST()</codeph> literal values where necessary,
+      rather than declaring <codeph>DECIMAL</codeph> columns with high precision for convenience.
+    </p>
+
+    <p>
+      To represent a very large or precise <codeph>DECIMAL</codeph> value as a literal, for example one that
+      contains more digits than can be represented by a <codeph>BIGINT</codeph> literal, use a quoted string or a
+      floating-point value for the number, and <codeph>CAST()</codeph> to the desired <codeph>DECIMAL</codeph>
+      type:
+    </p>
+
+<codeblock>insert into decimals_38_5 values (1), (2), (4), (8), (16), (1024), (32768), (65536), (1000000),
+  (cast("999999999999999999999999999999" as decimal(38,5))),
+  (cast(999999999999999999999999999999. as decimal(38,5)));
+</codeblock>
+
+    <ul>
+      <li>
+        <p> The result of the <codeph>SUM()</codeph> aggregate function on
+            <codeph>DECIMAL</codeph> values is promoted to a precision of 38,
+          with the same precision as the underlying column. Thus, the result can
+          represent the largest possible value at that particular precision. </p>
+      </li>
+
+      <li>
+        <p>
+          <codeph>STRING</codeph> columns, literals, or expressions can be converted to <codeph>DECIMAL</codeph> as
+          long as the overall number of digits and digits to the right of the decimal point fit within the
+          specified precision and scale for the declared <codeph>DECIMAL</codeph> type. By default, a
+          <codeph>DECIMAL</codeph> value with no specified scale or precision can hold a maximum of 9 digits of an
+          integer value. If there are more digits in the string value than are allowed by the
+          <codeph>DECIMAL</codeph> scale and precision, the result is <codeph>NULL</codeph>.
+        </p>
+        <p>
+          The following examples demonstrate how <codeph>STRING</codeph> values with integer and fractional parts
+          are represented when converted to <codeph>DECIMAL</codeph>. If the scale is 0, the number is treated
+          as an integer value with a maximum of <varname>precision</varname> digits. If the precision is greater than
+          0, the scale must be increased to account for the digits both to the left and right of the decimal point.
+          As the precision increases, output values are printed with additional trailing zeros after the decimal
+          point if needed. Any trailing zeros after the decimal point in the <codeph>STRING</codeph> value must fit
+          within the number of digits specified by the precision.
+        </p>
+<codeblock><![CDATA[[localhost:21000] > select cast('100' as decimal); -- Small integer value fits within 9 digits of scale.
++-----------------------------+
+| cast('100' as decimal(9,0)) |
++-----------------------------+
+| 100                         |
++-----------------------------+
+[localhost:21000] > select cast('100' as decimal(3,0)); -- Small integer value fits within 3 digits of scale.
++-----------------------------+
+| cast('100' as decimal(3,0)) |
++-----------------------------+
+| 100                         |
++-----------------------------+
+[localhost:21000] > select cast('100' as decimal(2,0)); -- 2 digits of scale is not enough!
++-----------------------------+
+| cast('100' as decimal(2,0)) |
++-----------------------------+
+| NULL                        |
++-----------------------------+
+[localhost:21000] > select cast('100' as decimal(3,1)); -- (3,1) = 2 digits left of the decimal point, 1 to the right. Not enough.
++-----------------------------+
+| cast('100' as decimal(3,1)) |
++-----------------------------+
+| NULL                        |
++-----------------------------+
+[localhost:21000] > select cast('100' as decimal(4,1)); -- 4 digits total, 1 to the right of the decimal point.
++-----------------------------+
+| cast('100' as decimal(4,1)) |
++-----------------------------+
+| 100.0                       |
++-----------------------------+
+[localhost:21000] > select cast('98.6' as decimal(3,1)); -- (3,1) can hold a 3 digit number with 1 fractional digit.
++------------------------------+
+| cast('98.6' as decimal(3,1)) |
++------------------------------+
+| 98.6                         |
++------------------------------+
+[localhost:21000] > select cast('98.6' as decimal(15,1)); -- Larger scale allows bigger numbers but still only 1 fractional digit.
++-------------------------------+
+| cast('98.6' as decimal(15,1)) |
++-------------------------------+
+| 98.6                          |
++-------------------------------+
+[localhost:21000] > select cast('98.6' as decimal(15,5)); -- Larger precision allows more fractional digits, outputs trailing zeros.
++-------------------------------+
+| cast('98.6' as decimal(15,5)) |
++-------------------------------+
+| 98.60000                      |
++-------------------------------+
+[localhost:21000] > select cast('98.60000' as decimal(15,1)); -- Trailing zeros in the string must fit within 'scale' digits (1 in this case).
++-----------------------------------+
+| cast('98.60000' as decimal(15,1)) |
++-----------------------------------+
+| NULL                              |
++-----------------------------------+
+]]>
+</codeblock>
+      </li>
+
+      <li>
+        Most built-in arithmetic functions such as <codeph>SIN()</codeph> and <codeph>COS()</codeph> continue to
+        accept only <codeph>DOUBLE</codeph> values because they are so commonly used in scientific context for
+        calculations of IEEE 954-compliant values. The built-in functions that accept and return
+        <codeph>DECIMAL</codeph> are:
+<!-- List from Skye: positive, negative, least, greatest, fnv_hash, if, nullif, zeroifnull, isnull, coalesce -->
+<!-- Nong had already told me about abs, ceil, floor, round, truncate -->
+        <ul>
+          <li>
+            <codeph>ABS()</codeph>
+          </li>
+
+          <li>
+            <codeph>CEIL()</codeph>
+          </li>
+
+          <li>
+            <codeph>COALESCE()</codeph>
+          </li>
+
+          <li>
+            <codeph>FLOOR()</codeph>
+          </li>
+
+          <li>
+            <codeph>FNV_HASH()</codeph>
+          </li>
+
+          <li>
+            <codeph>GREATEST()</codeph>
+          </li>
+
+          <li>
+            <codeph>IF()</codeph>
+          </li>
+
+          <li>
+            <codeph>ISNULL()</codeph>
+          </li>
+
+          <li>
+            <codeph>LEAST()</codeph>
+          </li>
+
+          <li>
+            <codeph>NEGATIVE()</codeph>
+          </li>
+
+          <li>
+            <codeph>NULLIF()</codeph>
+          </li>
+
+          <li>
+            <codeph>POSITIVE()</codeph>
+          </li>
+
+          <li>
+            <codeph>PRECISION()</codeph>
+          </li>
+
+          <li>
+            <codeph>ROUND()</codeph>
+          </li>
+
+          <li>
+            <codeph>SCALE()</codeph>
+          </li>
+
+          <li>
+            <codeph>TRUNCATE()</codeph>
+          </li>
+
+          <li>
+            <codeph>ZEROIFNULL()</codeph>
+          </li>
+        </ul>
+        See <xref href="impala_functions.xml#builtins"/> for details.
+      </li>
+
+      <li>
+        <p>
+          <codeph>BIGINT</codeph>, <codeph>INT</codeph>, <codeph>SMALLINT</codeph>, and <codeph>TINYINT</codeph>
+          values can all be cast to <codeph>DECIMAL</codeph>. The number of digits to the left of the decimal point
+          in the <codeph>DECIMAL</codeph> type must be sufficient to hold the largest value of the corresponding
+          integer type. Note that integer literals are treated as the smallest appropriate integer type, meaning
+          there is sometimes a range of values that require one more digit of <codeph>DECIMAL</codeph> scale than
+          you might expect. For integer values, the precision of the <codeph>DECIMAL</codeph> type can be zero; if
+          the precision is greater than zero, remember to increase the scale value by an equivalent amount to hold
+          the required number of digits to the left of the decimal point.
+        </p>
+        <p>
+          The following examples show how different integer types are converted to <codeph>DECIMAL</codeph>.
+        </p>
+<!-- According to Nong, it's a bug that so many integer digits can be converted to a DECIMAL
+     value with small (s,p) spec. So expect to re-do this example. -->
+<codeblock><![CDATA[[localhost:21000] > select cast(1 as decimal(1,0));
++-------------------------+
+| cast(1 as decimal(1,0)) |
++-------------------------+
+| 1                       |
++-------------------------+
+[localhost:21000] > select cast(9 as decimal(1,0));
++-------------------------+
+| cast(9 as decimal(1,0)) |
++-------------------------+
+| 9                       |
++-------------------------+
+[localhost:21000] > select cast(10 as decimal(1,0));
++--------------------------+
+| cast(10 as decimal(1,0)) |
++--------------------------+
+| 10                       |
++--------------------------+
+[localhost:21000] > select cast(10 as decimal(1,1));
++--------------------------+
+| cast(10 as decimal(1,1)) |
++--------------------------+
+| 10.0                     |
++--------------------------+
+[localhost:21000] > select cast(100 as decimal(1,1));
++---------------------------+
+| cast(100 as decimal(1,1)) |
++---------------------------+
+| 100.0                     |
++---------------------------+
+[localhost:21000] > select cast(1000 as decimal(1,1));
++----------------------------+
+| cast(1000 as decimal(1,1)) |
++----------------------------+
+| 1000.0                     |
++----------------------------+
+]]>
+</codeblock>
+      </li>
+
+      <li>
+        <p>
+          When a <codeph>DECIMAL</codeph> value is converted to any of the integer types, any fractional part is
+          truncated (that is, rounded towards zero):
+        </p>
+<codeblock><![CDATA[[localhost:21000] > create table num_dec_days (x decimal(4,1));
+[localhost:21000] > insert into num_dec_days values (1), (2), (cast(4.5 as decimal(4,1)));
+[localhost:21000] > insert into num_dec_days values (cast(0.1 as decimal(4,1))), (cast(.9 as decimal(4,1))), (cast(9.1 as decimal(4,1))), (cast(9.9 as decimal(4,1)));
+[localhost:21000] > select cast(x as int) from num_dec_days;
++----------------+
+| cast(x as int) |
++----------------+
+| 1              |
+| 2              |
+| 4              |
+| 0              |
+| 0              |
+| 9              |
+| 9              |
++----------------+
+]]>
+</codeblock>
+      </li>
+
+      <li>
+        <p>
+          You cannot directly cast <codeph>TIMESTAMP</codeph> or <codeph>BOOLEAN</codeph> values to or from
+          <codeph>DECIMAL</codeph> values. You can turn a <codeph>DECIMAL</codeph> value into a time-related
+          representation using a two-step process, by converting it to an integer value and then using that result
+          in a call to a date and time function such as <codeph>from_unixtime()</codeph>.
+        </p>
+<codeblock><![CDATA[[localhost:21000] > select from_unixtime(cast(cast(1000.0 as decimal) as bigint));
++-------------------------------------------------------------+
+| from_unixtime(cast(cast(1000.0 as decimal(9,0)) as bigint)) |
++-------------------------------------------------------------+
+| 1970-01-01 00:16:40                                         |
++-------------------------------------------------------------+
+[localhost:21000] > select now() + interval cast(x as int) days from num_dec_days; -- x is a DECIMAL column.
+
+[localhost:21000] > create table num_dec_days (x decimal(4,1));
+[localhost:21000] > insert into num_dec_days values (1), (2), (cast(4.5 as decimal(4,1)));
+[localhost:21000] > select now() + interval cast(x as int) days from num_dec_days; -- The 4.5 value is truncated to 4 and becomes '4 days'.
++--------------------------------------+
+| now() + interval cast(x as int) days |
++--------------------------------------+
+| 2014-05-13 23:11:55.163284000        |
+| 2014-05-14 23:11:55.163284000        |
+| 2014-05-16 23:11:55.163284000        |
++--------------------------------------+
+]]>
+</codeblock>
+      </li>
+
+      <li>
+        <p>
+          Because values in <codeph>INSERT</codeph> statements are checked rigorously for type compatibility, be
+          prepared to use <codeph>CAST()</codeph> function calls around literals, column references, or other
+          expressions that you are inserting into a <codeph>DECIMAL</codeph> column.
+        </p>
+      </li>
+    </ul>
+
+    <p conref="../shared/impala_common.xml#common/null_bad_numeric_cast"/>
+
+    <p>
+      <b>DECIMAL differences from integer and floating-point types:</b>
+    </p>
+
+    <p>
+      With the <codeph>DECIMAL</codeph> type, you are concerned with the number of overall digits of a number
+      rather than powers of 2 (as in <codeph>TINYINT</codeph>, <codeph>SMALLINT</codeph>, and so on). Therefore,
+      the limits with integral values of <codeph>DECIMAL</codeph> types fall around 99, 999, 9999, and so on rather
+      than 32767, 65535, 2
+      <sup>32</sup>
+      -1, and so on. For fractional values, you do not need to account for imprecise representation of the
+      fractional part according to the IEEE-954 standard (as in <codeph>FLOAT</codeph> and
+      <codeph>DOUBLE</codeph>). Therefore, when you insert a fractional value into a <codeph>DECIMAL</codeph>
+      column, you can compare, sum, query, <codeph>GROUP BY</codeph>, and so on that column and get back the
+      original values rather than some <q>close but not identical</q> value.
+    </p>
+
+    <p>
+      <codeph>FLOAT</codeph> and <codeph>DOUBLE</codeph> can cause problems or unexpected behavior due to inability
+      to precisely represent certain fractional values, for example dollar and cents values for currency. You might
+      find output values slightly different than you inserted, equality tests that do not match precisely, or
+      unexpected values for <codeph>GROUP BY</codeph> columns. <codeph>DECIMAL</codeph> can help reduce unexpected
+      behavior and rounding errors, at the expense of some performance overhead for assignments and comparisons.
+    </p>
+
+    <p>
+      <b>Literals and expressions:</b>
+      <ul>
+        <li>
+          <p>
+            When you use an integer literal such as <codeph>1</codeph> or <codeph>999</codeph> in a SQL statement,
+            depending on the context, Impala will treat it as either the smallest appropriate
+            <codeph>DECIMAL</codeph> type, or the smallest integer type (<codeph>TINYINT</codeph>,
+            <codeph>SMALLINT</codeph>, <codeph>INT</codeph>, or <codeph>BIGINT</codeph>). To minimize memory usage,
+            Impala prefers to treat the literal as the smallest appropriate integer type.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            When you use a floating-point literal such as <codeph>1.1</codeph> or <codeph>999.44</codeph> in a SQL
+            statement, depending on the context, Impala will treat it as either the smallest appropriate
+            <codeph>DECIMAL</codeph> type, or the smallest floating-point type (<codeph>FLOAT</codeph> or
+            <codeph>DOUBLE</codeph>). To avoid loss of accuracy, Impala prefers to treat the literal as a
+            <codeph>DECIMAL</codeph>.
+          </p>
+        </li>
+      </ul>
+    </p>
+
+    <p>
+      <b>Storage considerations:</b>
+    </p>
+
+    <ul>
+      <li>
+        Only the precision determines the storage size for <codeph>DECIMAL</codeph> values; the scale setting has
+        no effect on the storage size.
+      </li>
+
+      <li>
+        Text, RCFile, and SequenceFile tables all use ASCII-based formats. In these text-based file formats,
+        leading zeros are not stored, but trailing zeros are stored. In these tables, each <codeph>DECIMAL</codeph>
+        value takes up as many bytes as there are digits in the value, plus an extra byte if the decimal point is
+        present and an extra byte for negative values. Once the values are loaded into memory, they are represented
+        in 4, 8, or 16 bytes as described in the following list items. The on-disk representation varies depending
+        on the file format of the table.
+      </li>
+
+<!-- Next couple of points can be conref'ed with identical list bullets farther down under File Format Considerations. -->
+
+      <li>
+        Parquet and Avro tables use binary formats, In these tables, Impala stores each value in as few bytes as
+        possible
+<!-- 4, 8, or 16 bytes -->
+        depending on the precision specified for the <codeph>DECIMAL</codeph> column.
+        <ul>
+          <li>
+            In memory, <codeph>DECIMAL</codeph> values with precision of 9 or less are stored in 4 bytes.
+          </li>
+
+          <li>
+            In memory, <codeph>DECIMAL</codeph> values with precision of 10 through 18 are stored in 8 bytes.
+          </li>
+
+          <li>
+            In memory, <codeph>DECIMAL</codeph> values with precision greater than 18 are stored in 16 bytes.
+          </li>
+        </ul>
+      </li>
+    </ul>
+
+    <p conref="../shared/impala_common.xml#common/file_format_blurb"/>
+
+    <ul>
+      <li>
+        The <codeph>DECIMAL</codeph> data type can be stored in any of the file formats supported by Impala, as
+        described in <xref href="impala_file_formats.xml#file_formats"/>. Impala only writes to tables that use the
+        Parquet and text formats, so those formats are the focus for file format compatibility.
+      </li>
+
+      <li>
+        Impala can query Avro, RCFile, or SequenceFile tables containing <codeph>DECIMAL</codeph> columns, created
+        by other Hadoop components, on CDH 5 only.
+      </li>
+
+      <li>
+        You can use <codeph>DECIMAL</codeph> columns in Impala tables that are mapped to HBase tables. Impala can
+        query and insert into such tables.
+      </li>
+
+      <li>
+        Text, RCFile, and SequenceFile tables all use ASCII-based formats. In these tables, each
+        <codeph>DECIMAL</codeph> value takes up as many bytes as there are digits in the value, plus an extra byte
+        if the decimal point is present. The binary format of Parquet or Avro files offers more compact storage for
+        <codeph>DECIMAL</codeph> columns.
+      </li>
+
+      <li>
+        Parquet and Avro tables use binary formats, In these tables, Impala stores each value in 4, 8, or 16 bytes
+        depending on the precision specified for the <codeph>DECIMAL</codeph> column.
+      </li>
+
+      <li>
+        Parquet files containing <codeph>DECIMAL</codeph> columns are not expected to be readable under CDH 4. See
+        the <b>Compatibility</b> section for details.
+      </li>
+    </ul>
+
+    <p>
+      <b>UDF considerations:</b> When writing a C++ UDF, use the <codeph>DecimalVal</codeph> data type defined in
+      <filepath>/usr/include/impala_udf/udf.h</filepath>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/partitioning_blurb"/>
+
+    <p>
+      You can use a <codeph>DECIMAL</codeph> column as a partition key. Doing so provides a better match between
+      the partition key values and the HDFS directory names than using a <codeph>DOUBLE</codeph> or
+      <codeph>FLOAT</codeph> partitioning column:
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/schema_evolution_blurb"/>
+
+    <ul>
+      <li>
+        For text-based formats (text, RCFile, and SequenceFile tables), you can issue an <codeph>ALTER TABLE ...
+        REPLACE COLUMNS</codeph> statement to change the precision and scale of an existing
+        <codeph>DECIMAL</codeph> column. As long as the values in the column fit within the new precision and
+        scale, they are returned correctly by a query. Any values that do not fit within the new precision and
+        scale are returned as <codeph>NULL</codeph>, and Impala reports the conversion error. Leading zeros do not
+        count against the precision value, but trailing zeros after the decimal point do.
+<codeblock><![CDATA[[localhost:21000] > create table text_decimals (x string);
+[localhost:21000] > insert into text_decimals values ("1"), ("2"), ("99.99"), ("1.234"), ("000001"), ("1.000000000");
+[localhost:21000] > select * from text_decimals;
++-------------+
+| x           |
++-------------+
+| 1           |
+| 2           |
+| 99.99       |
+| 1.234       |
+| 000001      |
+| 1.000000000 |
++-------------+
+[localhost:21000] > alter table text_decimals replace columns (x decimal(4,2));
+[localhost:21000] > select * from text_decimals;
++-------+
+| x     |
++-------+
+| 1.00  |
+| 2.00  |
+| 99.99 |
+| NULL  |
+| 1.00  |
+| NULL  |
++-------+
+ERRORS:
+Backend 0:Error converting column: 0 TO DECIMAL(4, 2) (Data is: 1.234)
+file: hdfs://127.0.0.1:8020/user/hive/warehouse/decimal_testing.db/text_decimals/634d4bd3aa0
+e8420-b4b13bab7f1be787_56794587_data.0
+record: 1.234
+Error converting column: 0 TO DECIMAL(4, 2) (Data is: 1.000000000)
+file: hdfs://127.0.0.1:8020/user/hive/warehouse/decimal_testing.db/text_decimals/cd40dc68e20
+c565a-cc4bd86c724c96ba_311873428_data.0
+record: 1.000000000
+]]>
+</codeblock>
+      </li>
+
+      <li>
+        For binary formats (Parquet and Avro tables), although an <codeph>ALTER TABLE ... REPLACE COLUMNS</codeph>
+        statement that changes the precision or scale of a <codeph>DECIMAL</codeph> column succeeds, any subsequent
+        attempt to query the changed column results in a fatal error. (The other columns can still be queried
+        successfully.) This is because the metadata about the columns is stored in the data files themselves, and
+        <codeph>ALTER TABLE</codeph> does not actually make any updates to the data files. If the metadata in the
+        data files disagrees with the metadata in the metastore database, Impala cancels the query.
+      </li>
+    </ul>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>CREATE TABLE t1 (x DECIMAL, y DECIMAL(5,2), z DECIMAL(25,0));
+INSERT INTO t1 VALUES (5, 99.44, 123456), (300, 6.7, 999999999);
+SELECT x+y, ROUND(y,1), z/98.6 FROM t1;
+SELECT CAST(1000.5 AS DECIMAL);
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/decimal_no_stats"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/partitioning_good"/> -->
+
+    <p conref="../shared/impala_common.xml#common/hbase_ok"/>
+
+    <p conref="../shared/impala_common.xml#common/parquet_ok"/>
+
+    <p conref="../shared/impala_common.xml#common/text_bulky"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/compatibility_blurb"/> -->
+
+<!-- <p conref="../shared/impala_common.xml#common/internals_blurb"/> -->
+
+<!-- <p conref="../shared/impala_common.xml#common/added_in_20"/> -->
+
+    <p conref="../shared/impala_common.xml#common/column_stats_constant"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_literals.xml#numeric_literals"/>, <xref href="impala_tinyint.xml#tinyint"/>,
+      <xref href="impala_smallint.xml#smallint"/>, <xref href="impala_int.xml#int"/>,
+      <xref href="impala_bigint.xml#bigint"/>, <xref href="impala_decimal.xml#decimal"/>,
+      <xref href="impala_math_functions.xml#math_functions"/> (especially <codeph>PRECISION()</codeph> and
+      <codeph>SCALE()</codeph>)
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_default_order_by_limit.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_default_order_by_limit.xml b/docs/topics/impala_default_order_by_limit.xml
new file mode 100644
index 0000000..94f6899
--- /dev/null
+++ b/docs/topics/impala_default_order_by_limit.xml
@@ -0,0 +1,37 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="obwl" id="default_order_by_limit">
+
+  <title>DEFAULT_ORDER_BY_LIMIT Query Option</title>
+  <titlealts audience="PDF"><navtitle>DEFAULT_ORDER_BY_LIMIT</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p conref="../shared/impala_common.xml#common/obwl_query_options"/>
+
+    <p rev="1.4.0">
+      Prior to Impala 1.4.0, Impala queries that use the <codeph><xref href="impala_order_by.xml#order_by">ORDER
+      BY</xref></codeph> clause must also include a
+      <codeph><xref href="impala_limit.xml#limit">LIMIT</xref></codeph> clause, to avoid accidentally producing
+      huge result sets that must be sorted. Sorting a huge result set is a memory-intensive operation. In Impala
+      1.4.0 and higher, Impala uses a temporary disk work area to perform the sort if that operation would
+      otherwise exceed the Impala memory limit on a particular host.
+    </p>
+
+    <p>
+      <b>Type: numeric</b>
+    </p>
+
+    <p>
+      <b>Default:</b> -1 (no default limit)
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_delegation.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_delegation.xml b/docs/topics/impala_delegation.xml
new file mode 100644
index 0000000..0d59761
--- /dev/null
+++ b/docs/topics/impala_delegation.xml
@@ -0,0 +1,88 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.2" id="delegation">
+
+  <title>Configuring Impala Delegation for Hue and BI Tools</title>
+
+  <prolog>
+    <metadata>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Authentication"/>
+      <data name="Category" value="Delegation"/>
+      <data name="Category" value="Hue"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+<!--
+      When users connect to Impala directly through the <cmdname>impala-shell</cmdname> interpreter, the Sentry
+      authorization framework determines what actions they can take and what data they can see.
+-->
+      When users submit Impala queries through a separate application, such as Hue or a business intelligence tool,
+      typically all requests are treated as coming from the same user. In Impala 1.2 and higher, authentication is
+      extended by a new feature that allows applications to pass along credentials for the users that connect to
+      them (known as <q>delegation</q>), and issue Impala queries with the privileges for those users. Currently,
+      the delegation feature is available only for Impala queries submitted through application interfaces such as
+      Hue and BI tools; for example, Impala cannot issue queries using the privileges of the HDFS user.
+    </p>
+
+    <p>
+      The delegation feature is enabled by a startup option for <cmdname>impalad</cmdname>:
+      <codeph>--authorized_proxy_user_config</codeph>. When you specify this option, users whose names you specify
+      (such as <codeph>hue</codeph>) can delegate the execution of a query to another user. The query runs with the
+      privileges of the delegated user, not the original user such as <codeph>hue</codeph>. The name of the
+      delegated user is passed using the HiveServer2 configuration property <codeph>impala.doas.user</codeph>.
+    </p>
+
+    <p>
+      You can specify a list of users that the application user can delegate to, or <codeph>*</codeph> to allow a
+      superuser to delegate to any other user. For example:
+    </p>
+
+<codeblock>impalad --authorized_proxy_user_config 'hue=user1,user2;admin=*' ...</codeblock>
+
+    <note>
+      Make sure to use single quotes or escape characters to ensure that any <codeph>*</codeph> characters do not
+      undergo wildcard expansion when specified in command-line arguments.
+    </note>
+
+    <p>
+      See <xref href="impala_config_options.xml#config_options"/> for details about adding or changing
+      <cmdname>impalad</cmdname> startup options. See
+      <xref href="http://blog.cloudera.com/blog/2013/07/how-hiveserver2-brings-security-and-concurrency-to-apache-hive/" scope="external" format="html">this
+      Cloudera blog post</xref> for background information about the delegation capability in HiveServer2.
+    </p>
+
+    <p>
+      To set up authentication for the delegated users:
+    </p>
+
+    <ul>
+      <li>
+        <p>
+          On the server side, configure either user/password authentication through LDAP, or Kerberos
+          authentication, for all the delegated users. See <xref href="impala_ldap.xml#ldap"/> or
+          <xref href="impala_kerberos.xml#kerberos"/> for details.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          On the client side, follow the instructions in the <q>Using User Name and Password</q> section in the
+          <xref href="http://www.cloudera.com/content/cloudera-content/cloudera-docs/Connectors/PDF/Cloudera-ODBC-Driver-for-Impala-Install-Guide.pdf" scope="external" format="pdf">ODBC
+          driver installation guide</xref>. Then search for <q>delegation</q> in that same installation guide to
+          learn about the <uicontrol>Delegation UID</uicontrol> field and <codeph>DelegationUID</codeph> configuration keyword to enable the delegation feature for
+          ODBC-based BI tools.
+        </p>
+      </li>
+    </ul>
+
+  </conbody>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_delete.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_delete.xml b/docs/topics/impala_delete.xml
new file mode 100644
index 0000000..5715cc7
--- /dev/null
+++ b/docs/topics/impala_delete.xml
@@ -0,0 +1,65 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="delete">
+
+  <title>DELETE Statement (<keyword keyref="impala28"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>DELETE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Kudu"/>
+      <data name="Category" value="ETL"/>
+      <data name="Category" value="Ingest"/>
+      <data name="Category" value="DML"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">DELETE statement</indexterm>
+      Deletes one or more rows from a Kudu table.
+      Although deleting a single row or a range of rows would be inefficient for tables using HDFS
+      data files, Kudu is able to perform this operation efficiently. Therefore, this statement
+      only works for Impala tables that use the Kudu storage engine.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>
+</codeblock>
+
+    <p rev="kudu">
+      Normally, a <codeph>DELETE</codeph> operation for a Kudu table fails if
+      some partition key columns are not found, due to their being deleted or changed
+      by a concurrent <codeph>UPDATE</codeph> or <codeph>DELETE</codeph> operation.
+      Specify <codeph>DELETE IGNORE <varname>rest_of_statement</varname></codeph> to
+      make the <codeph>DELETE</codeph> continue in this case. The rows with the nonexistent
+      duplicate partition key column values are not removed.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/dml_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/sync_ddl_blurb"/>
+
+    <note conref="../shared/impala_common.xml#common/compute_stats_next"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_kudu.xml#impala_kudu"/>
+    </p>
+
+  </conbody>
+
+</concept>


[32/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_errata.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_errata.xml b/docs/topics/impala_errata.xml
new file mode 100644
index 0000000..c7ad9c0
--- /dev/null
+++ b/docs/topics/impala_errata.xml
@@ -0,0 +1,1378 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="errata">
+  <title>Impala Documentation Errata</title>
+  <titlealts audience="PDF"><navtitle>Doc Errata</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Release Notes" />
+      <data name="Category" value="New Features" />
+      <data name="Category" value="What's New" />
+      <data name="Category" value="Fixed Issues" />
+      <data name="Category" value="Incompatible Changes" />
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      This page shows changes in the Impala documentation.
+      These different views are experiments to see what kind(s) of
+      representation are most helpful.
+      Some changes are tied to a release.
+      Other improvements are the result of a particular JIRA issue and have
+      been backported to multiple releases where appropriate.
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+    <concept id="by_release">
+      <title>Changes by Release</title>
+      <conbody>
+        <p>
+          This section shows recent releases of CDH and corresponding Impala releases,
+          and the documentation pages that have substantial changes for each release.
+        </p>
+        <p outputclass="toc inpage"/>
+      </conbody>
+
+      <concept id="release_2_6_0">
+        <title>CDH 5.8 / Impala 2.6.0</title>
+        <conbody>
+       <ul>
+         <li><xref outputclass="release_2_6_0" href="impala_alter_table.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_breakpad.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_create_database.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_create_table.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_complex_types.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_drop_database.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_drop_table.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_perf_stats.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_insert.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_reserved_words.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_load_data.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_parquet_annotate_strings_utf8.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_parquet_fallback_schema_resolution.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_runtime_bloom_filter_size.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_runtime_filter_max_size.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_runtime_filter_min_size.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_runtime_filter_mode.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_runtime_filtering.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_s3_skip_insert_staging.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_select.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_s3.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_txtfile.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_avro.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_parquet.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_rcfile.xml"/></li>
+         <li><xref outputclass="release_2_6_0" href="impala_seqfile.xml"/></li>
+       </ul>
+
+</conbody>
+</concept>
+
+      <concept id="release_2_5_0">
+        <title>CDH 5.7 / Impala 2.5.0</title>
+        <conbody>
+       <ul>
+         <li><xref outputclass="release_2_5_0" href="impala_admission.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_count.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_create_database.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_create_function.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_create_table.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_describe.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_disable_row_runtime_filtering.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_disable_streaming_preaggregations.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_drop_function.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_datetime_functions.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_misc_functions.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_reserved_words.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_string_functions.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_udf.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_joins.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_load_data.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_max.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_max_num_runtime_filters.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_min.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_optimize_partition_key_scans.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_partitioning.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_runtime_bloom_filter_size.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_runtime_filter_max_size.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_runtime_filter_min_size.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_runtime_filter_mode.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_runtime_filter_wait_time_ms.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_shell_running_commands.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_runtime_filtering.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_scan_node_codegen_threshold.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_schedule_random_replica.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_set.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_show.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_operators.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_scalability.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_processes.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_truncate_table.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_perf_hdfs_caching.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_impala_shell.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_parquet.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_shell_commands.xml"/></li>
+         <li><xref outputclass="release_2_5_0" href="impala_shell_options.xml"/></li>
+       </ul>
+
+</conbody>
+</concept>
+
+      <concept id="release_2_3_0">
+        <title>CDH 5.5 / Impala 2.3.0</title>
+        <conbody>
+       <ul>
+         <li><xref outputclass="release_2_3_0" href="impala_alter_table.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_array.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_avg.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_compute_stats.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_count.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_create_database.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_create_function.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_create_table.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_create_view.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_complex_types.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_jdbc.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_odbc.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_ssl.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_describe.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_drop_database.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_drop_table.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_exec_single_node_rows_threshold.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_authorization.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_grant.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_group_by.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_insert.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_bit_functions.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_datetime_functions.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_math_functions.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_reserved_words.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_string_functions.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_conversion_functions.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_udf.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_joins.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_live_progress.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_live_summary.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_load_data.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_map.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_max.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_min.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_ndv.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_order_by.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_aliases.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_tables.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_views.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_partitioning.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_refresh.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_revoke.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_select.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_set.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_operators.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_struct.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_sum.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_subqueries.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_truncate_table.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_hbase.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_txtfile.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_avro.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_parquet.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_rcfile.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_seqfile.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_shell_commands.xml"/></li>
+         <li><xref outputclass="release_2_3_0" href="impala_shell_options.xml"/></li>
+       </ul>
+
+</conbody>
+</concept>
+
+      <concept id="release_2_2_3">
+        <title>CDH 5.4 / Impala 2.2.3</title>
+        <conbody>
+       <ul>
+         <li><xref outputclass="release_2_2_3" href="impala_datetime_functions.xml"/></li>
+         <li><xref outputclass="release_2_2_3" href="impala_parquet_file_size.xml"/></li>
+         <li><xref outputclass="release_2_2_3" href="impala_isilon.xml"/></li>
+       </ul>
+
+</conbody>
+</concept>
+
+      <concept id="release_2_2_0">
+        <title>CDH 5.4 / Impala 2.2.0</title>
+        <conbody>
+       <ul>
+         <li><xref outputclass="release_2_2_0" href="impala_alter_table.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_alter_view.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_incompatible_changes.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_intro.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_compute_stats.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_create_database.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_create_table.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_create_view.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_ddl.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_dml.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_drop_database.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_drop_table.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_explain.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_insert.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_invalidate_metadata.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_conditional_functions.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_datetime_functions.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_faq.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_install.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_math_functions.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_conversion_functions.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_load_data.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_disk_space.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_tables.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_partitioning.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_refresh.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_select.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_show.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_scalability.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_timestamp.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_truncate_table.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_performance.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_explain_plan.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_perf_hdfs_caching.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_logging.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_s3.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_txtfile.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_parquet.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_lineage.xml"/></li>
+         <li><xref outputclass="release_2_2_0" href="impala_new_features.xml"/></li>
+       </ul>
+
+</conbody>
+</concept>
+
+      <concept id="release_2_1_0">
+        <title>CDH 5.3 / Impala 2.1.0</title>
+        <conbody>
+       <ul>
+         <li><xref outputclass="release_2_1_0" href="impala_compute_stats.xml"/></li>
+         <li><xref outputclass="release_2_1_0" href="impala_drop_stats.xml"/></li>
+         <li><xref outputclass="release_2_1_0" href="impala_exec_single_node_rows_threshold.xml"/></li>
+         <li><xref outputclass="release_2_1_0" href="impala_reserved_words.xml"/></li>
+         <li><xref outputclass="release_2_1_0" href="impala_ports.xml"/></li>
+         <li><xref outputclass="release_2_1_0" href="impala_txtfile.xml"/></li>
+       </ul>
+
+</conbody>
+</concept>
+
+      <concept id="release_2_0_1">
+        <title>CDH 5.2 / Impala 2.0.1</title>
+        <conbody>
+       <ul>
+         <li><xref outputclass="release_2_0_1" href="impala_perf_stats.xml"/></li>
+         <li><xref outputclass="release_2_0_1" href="impala_math_functions.xml"/></li>
+       </ul>
+
+</conbody>
+</concept>
+
+      <concept id="release_2_0_0">
+        <title>CDH 5.2 / Impala 2.0.0</title>
+        <conbody>
+       <ul>
+         <li><xref outputclass="release_2_0_0" href="impala_appx_count_distinct.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_avg.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_incompatible_changes.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_char.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_compression_codec.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_count.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_create_function.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_create_table.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_ddl.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_disable_unsafe_spills.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_exec_single_node_rows_threshold.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_kerberos.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_grant.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_having.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_file_formats.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_analytic_functions.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_functions.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_datetime_functions.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_faq.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_misc_functions.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_perf_cookbook.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_prereqs.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_reserved_words.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_string_functions.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_udf.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_joins.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_limit.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_literals.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_max.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_min.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_ndv.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_query_timeout_s.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_hints.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_query_options.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_revoke.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_resource_management.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_set.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_show.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_operators.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_stddev.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_string.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_sum.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_timeouts.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_subqueries.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_txtfile.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_varchar.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_variance.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_shell_commands.xml"/></li>
+         <li><xref outputclass="release_2_0_0" href="impala_shell_options.xml"/></li>
+       </ul>
+
+</conbody>
+</concept>
+</concept>
+
+  <concept id="by_jira">
+    <title>Changes by JIRA Issue</title>
+    <conbody>
+      <p>
+        This section lists Impala-related issues from the Impala issue-tracking system.
+        These items represent new features, fixes for documentation bugs, and workarounds
+        and notices for code bugs or limitations.
+        The same text may be reused in identical form on multiple pages.
+        Use this section to cross-check the documentation changes for feature requests and
+        problem reports, and to see the details in the issue-tracking system for features
+        and fixes that are only mentioned briefly in the documentation, such as performance
+        improvements that do not require any user tuning actions.
+      </p>
+    </conbody>
+
+  <concept id="issue_IMPALA-3732">
+    <title>IMPALA-3732: Avro scanner can break on strings &gt;= 2GB</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3732">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3732" href="impala_char.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3732" href="impala_varchar.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3732" href="impala_avro.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3732" href="impala_string.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3679">
+    <title>IMPALA-3679: Zero copy reads don&#39;t work with encryption zones</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3679">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3679" href="impala_perf_hdfs_caching.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3662">
+    <title>IMPALA-3662: Parquet scanner memory increase after IMPALA-2736</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3662">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3662" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3581">
+    <title>IMPALA-3581: Change Default Minidump Location to /var/log/$daemon</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3581">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3581" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3558">
+    <title>IMPALA-3558: DROP TABLE PURGE on S3A table may not delete externally written files</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3558">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3558" href="impala_s3.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3558" href="impala_drop_table.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3554">
+    <title>IMPALA-3554: Use catalog&#39;s principal to talk to sentry in kerberized Impala clusters</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3554">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3554" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3554" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3490">
+    <title>IMPALA-3490: Add Flag to Cap Maximum Minidump Size</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3490">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3490" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3480">
+    <title>IMPALA-3480: Add query options to set min / max filter sizes</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3480">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3480" href="impala_runtime_filter_min_size.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3480" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3480" href="impala_runtime_filter_max_size.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3480" href="impala_runtime_filtering.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3471">
+    <title>IMPALA-3471: TopN should be able to spill</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3471">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3471" href="impala_scalability.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3471" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3453">
+    <title>IMPALA-3453: S3 : Uneven split sizes are generated for Parquet causing execution skew</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3453">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3453" href="impala_s3.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3453" href="impala_avro.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3453" href="impala_parquet.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3453" href="impala_select.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3453" href="impala_seqfile.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3453" href="impala_txtfile.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3453" href="impala_rcfile.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3452">
+    <title>IMPALA-3452: S3: Disable Impala staging for INSERTs via flag for speedup</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3452">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3452" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3452" href="impala_s3_skip_insert_staging.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3452" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3439">
+    <title>IMPALA-3439: Error choosing function signature in ROUND()</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3439">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3439" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3439" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3397">
+    <title>IMPALA-3397: Support SOURCE &lt;file&gt; in shell</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3397">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3397" href="impala_shell_running_commands.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3397" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3397" href="impala_shell_commands.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3375">
+    <title>IMPALA-3375: Investigate expensive calls to tc_delete_nothrow &amp; tc_new during TopN for 2x speedup.</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3375">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3375" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3369">
+    <title>IMPALA-3369: Add a variant of alter table to manually add column stats</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3369">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3369" href="impala_alter_table.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3369" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3369" href="impala_perf_stats.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3333">
+    <title>IMPALA-3333: Enable global runtime filtering by default</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3333">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3333" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3333" href="impala_runtime_filtering.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3232">
+    <title>IMPALA-3232: Allow table refs in subqueries to refer to WITH-clause view in parent block.</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3232">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3232" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3232" href="impala_operators.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3199">
+    <title>IMPALA-3199: Enable Admission Control by default (without limits)</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3199">Issue</xref></p>
+     <ul>
+<!--       <li><xref outputclass="issue_IMPALA-3199" href="admin_impala_admission_control.xml"/></li> -->
+       <li><xref outputclass="issue_IMPALA-3199" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3199" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3163">
+    <title>IMPALA-3163: DECIMAL casting to a TIMESTAMP doesn&#39;t keep precision</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3163">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3163" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3163" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3155">
+    <title>IMPALA-3155: CHAR expression in case statement is implicitly cast to string</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3155">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3155" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3155" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3095">
+    <title>IMPALA-3095: Allow additional Kerberos users to be authorized to access internal APIs</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3095">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3095" href="impala_kerberos.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3095" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3092">
+    <title>IMPALA-3092: Cannot query tables after add columns for AVRO table</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3092">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3092" href="impala_alter_table.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3092" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3077">
+    <title>IMPALA-3077: Runtime filters should be retained even when spilling</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3077">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3077" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3054">
+    <title>IMPALA-3054: Runtime filters are not disabled when spilling in a rare case</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3054">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3054" href="impala_scalability.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3054" href="impala_partitioning.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3054" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3054" href="impala_runtime_filtering.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3046">
+    <title>IMPALA-3046: Allow per-row filters to evaluate non-SlotRef exprs</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3046">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3046" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3044">
+    <title>IMPALA-3044: Fix admission control default value for max requests</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3044">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3044" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3039">
+    <title>IMPALA-3039: Restrict the number of runtime filters generated </title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3039">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3039" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3008">
+    <title>IMPALA-3008: Improve printing of filter routing table</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3008">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3008" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3007">
+    <title>IMPALA-3007: Use planner cardinality estimates when sizing bloom filters</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3007">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3007" href="impala_runtime_bloom_filter_size.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3007" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3007" href="impala_incompatible_changes.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-3007" href="impala_runtime_filtering.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-3001">
+    <title>IMPALA-3001: Improve usability of RUNTIME_FILTER query options</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3001">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-3001" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2956">
+    <title>IMPALA-2956: Filters should be able to target multiple scans</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2956">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2956" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2867">
+    <title>IMPALA-2867: Load data inpath chokes on impala staging directories</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2867">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2867" href="impala_load_data.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2867" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2844">
+    <title>IMPALA-2844: Select count(*) from RCFileTable throw NotImplementedException in Impala 2.3.0</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2844">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2844" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2844" href="impala_avro.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2844" href="impala_complex_types.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2844" href="impala_seqfile.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2844" href="impala_txtfile.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2844" href="impala_rcfile.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2843">
+    <title>IMPALA-2843: Impala should read java udfs created from Hive</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2843">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2843" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2843" href="impala_create_function.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2843" href="impala_udf.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2843" href="impala_drop_function.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2835">
+    <title>IMPALA-2835: Hive/Impala inconsistency with parquet.column.index.access=false</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2835">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2835" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2835" href="impala_parquet.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2835" href="impala_parquet_fallback_schema_resolution.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2835" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2749">
+    <title>IMPALA-2749: Impala returns NULL when decimal and double values are multiplied in a specific order</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2749">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2749" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2749" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2736">
+    <title>IMPALA-2736: Column-wise value materialisation in Parquet scanner</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2736">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2736" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2728">
+    <title>IMPALA-2728: TestTpchMemLimitError.test_low_mem_limit_q21 exceeds mem_limit of 700M</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2728">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2728" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2696">
+    <title>IMPALA-2696: Simple scheduler always uses the same replica which can create hot spots</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2696">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2696" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2688">
+    <title>IMPALA-2688: Group by on Decimal data type doesn&#39;t get CodeGened </title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2688">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2688" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2686">
+    <title>IMPALA-2686: Implement google BreakPad based crash handler for impala.</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2686">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2686" href="impala_breakpad.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2686" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2681">
+    <title>IMPALA-2681: Improve Decimal arithmetic performance by using a cheaper overflow check  </title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2681">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2681" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2660">
+    <title>IMPALA-2660: Respect auth_to_local rules from hdfs configs (core-site.xml)</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2660">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2660" href="impala_kerberos.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2660" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2660" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2641">
+    <title>IMPALA-2641: Add IF EXISTS clause to TRUNCATE TABLE statement</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2641">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2641" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2641" href="impala_truncate_table.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2599">
+    <title>IMPALA-2599: Pseudo-random sleep before acquiring kerberos ticket possibly not really pseudo-random.</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2599">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2599" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2560">
+    <title>IMPALA-2560: Large memory consumption on coordinator node when handling a large number of partitions due to redundancy in TDescriptorTable</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2560">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2560" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2552">
+    <title>IMPALA-2552: Runtime filter forwarding between operators</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2552">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2552" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2538">
+    <title>IMPALA-2538: Per-pool default query memory limits</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2538">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2538" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2538" href="impala_admission.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2499">
+    <title>IMPALA-2499: Min/max values on partition columns</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2499">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2499" href="impala_partitioning.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2499" href="impala_optimize_partition_key_scans.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2499" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2419">
+    <title>IMPALA-2419: Extend Probe filter implementation</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2419">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2419" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2382">
+    <title>IMPALA-2382: Impala unable to read Java based UDFs that return a standard  datatype like Float or String(This works in hive)</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2382">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2382" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2336">
+    <title>IMPALA-2336: Failure to execute SQL scripts with trailing comments (including comment-only scripts)</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2336">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2336" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2257">
+    <title>IMPALA-2257: Text delimiter breaks data loading build</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2257">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2257" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2251">
+    <title>IMPALA-2251: Impala allows creation of text tables with same row and field delimiter</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2251">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2251" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2233">
+    <title>IMPALA-2233: Impala should not choose a function signature that results in loss of precision</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2233">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2233" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2204">
+    <title>IMPALA-2204: Underscore in where does not work for multi-line text </title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2204">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2204" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2196">
+    <title>IMPALA-2196: Add support for DESCRIBE DATABASE similar to Hive</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2196">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2196" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2196" href="impala_describe.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2180">
+    <title>IMPALA-2180:  IMPALA-2180: Extend SET command to allow setting variables in Impala Shell.</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2180">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2180" href="impala_set.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2180" href="impala_shell_running_commands.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2180" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2180" href="impala_shell_commands.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2179">
+    <title>IMPALA-2179:  IMPALA-2179: Extend Impala shell to allow passing variables through the command line</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2179">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2179" href="impala_shell_running_commands.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2179" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2179" href="impala_impala_shell.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2147">
+    <title>IMPALA-2147: Support for null-safe equal/IS [NOT] DISTINCT FROM</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2147">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2147" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2130">
+    <title>IMPALA-2130: hdfs-parquet-scanner&#39;s verification of file version is not correct</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2130">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2130" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2111">
+    <title>IMPALA-2111: Parquet TIMESTAMP is in milliseconds</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2111">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2111" href="impala_timestamp.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2111" href="impala_parquet.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2111" href="impala_bigint.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2070">
+    <title>IMPALA-2070: Include the database comment when showing databases</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2070">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2070" href="impala_show.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2070" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2070" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-2069">
+    <title>IMPALA-2069: Cannot write string column to parquet with UTF-8 annotation</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-2069">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-2069" href="impala_parquet_annotate_strings_utf8.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2069" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-2069" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1934">
+    <title>IMPALA-1934: impala-shell to support reading password from command line when LDAP authentication is used</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1934">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1934" href="impala_shell_options.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1934" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1881">
+    <title>IMPALA-1881: Maximize data locality when scanning Parquet files with multiple row groups.</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1881">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1881" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1878">
+    <title>IMPALA-1878: Support INSERT and LOAD DATA to S3</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1878">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1878" href="impala_s3.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1878" href="impala_alter_table.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1878" href="impala_create_table.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1878" href="impala_load_data.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1878" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1878" href="impala_drop_database.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1878" href="impala_insert.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1878" href="impala_create_database.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1878" href="impala_drop_table.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1787">
+    <title>IMPALA-1787: Case-insensitivity modifier for regular expressions</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1787">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1787" href="impala_string_functions.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1787" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1772">
+    <title>IMPALA-1772: Additional date / time fns</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1772">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1772" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1771">
+    <title>IMPALA-1771: Hyperbolic trig fns etc.</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1771">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1771" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1771" href="impala_math_functions.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1755">
+    <title>IMPALA-1755: Disabling codegen on parquet scanner causes some queries&#39; performance to regress</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1755">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1755" href="impala_scan_node_codegen_threshold.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1755" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1748">
+    <title>IMPALA-1748: Provide support for registering permanent udfs</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1748">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1748" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1748" href="impala_create_function.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1748" href="impala_udf.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1748" href="impala_incompatible_changes.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1748" href="impala_processes.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1748" href="impala_drop_function.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1740">
+    <title>IMPALA-1740: Impala cannot deal with skip.header.line.count</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1740">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1740" href="impala_new_features.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1740" href="impala_txtfile.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1675">
+    <title>IMPALA-1675: Timestamp: Adding/subtracting very large time intervals to timestamps produces incorrect result</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1675">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1675" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1651">
+    <title>IMPALA-1651: CREATE TABLE LIKE should not copy caching directives of the source table.</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1651">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1651" href="impala_incompatible_changes.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1599">
+    <title>IMPALA-1599: Improve query start-up time with many fragment instances</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1599">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1599" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1588">
+    <title>IMPALA-1588: Cache HDFS file handle to avoid repeated hdfs fopen call</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1588">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1588" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1568">
+    <title>IMPALA-1568: Find / repair missing partitions command</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1568">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1568" href="impala_alter_table.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1480">
+    <title>IMPALA-1480: Slow DDL statements for tables with large number of partitions</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1480">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1480" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1477">
+    <title>IMPALA-1477: Add UUID function</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1477">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1477" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1305">
+    <title>IMPALA-1305: Improve performance of high cardinality / low reduction aggs</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1305">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1305" href="impala_disable_streaming_preaggregations.xml"/></li>
+       <li><xref outputclass="issue_IMPALA-1305" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1168">
+    <title>IMPALA-1168: Finer grain Admission Control timeout control</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1168">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1168" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1139">
+    <title>IMPALA-1139: Implement TRUNCATE TABLE statement</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1139">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1139" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1079">
+    <title>IMPALA-1079: Impalad crashes while trying to run tpcds-q34</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1079">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1079" href="impala_shell_options.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  <concept id="issue_IMPALA-1067">
+    <title>IMPALA-1067: Passing Arguments through Impala Shell</title>
+    <conbody>
+      <p><xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-1067">Issue</xref></p>
+     <ul>
+       <li><xref outputclass="issue_IMPALA-1067" href="impala_new_features.xml"/></li>
+     </ul>
+
+</conbody>
+</concept>
+
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_exec_single_node_rows_threshold.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_exec_single_node_rows_threshold.xml b/docs/topics/impala_exec_single_node_rows_threshold.xml
new file mode 100644
index 0000000..a7cf102
--- /dev/null
+++ b/docs/topics/impala_exec_single_node_rows_threshold.xml
@@ -0,0 +1,96 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="2.0.0" id="exec_single_node_rows_threshold">
+
+  <title>EXEC_SINGLE_NODE_ROWS_THRESHOLD Query Option (<keyword keyref="impala21"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>EXEC_SINGLE_NODE_ROWS_THRESHOLD</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Scalability"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.0.0">
+      <indexterm audience="Cloudera">EXEC_SINGLE_NODE_ROWS_THRESHOLD query option</indexterm>
+      This setting controls the cutoff point (in terms of number of rows scanned) below which Impala treats a query
+      as a <q>small</q> query, turning off optimizations such as parallel execution and native code generation. The
+      overhead for these optimizations is applicable for queries involving substantial amounts of data, but it
+      makes sense to skip them for queries involving tiny amounts of data. Reducing the overhead for small queries
+      allows Impala to complete them more quickly, keeping YARN resources, admission control slots, and so on
+      available for data-intensive queries.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>SET EXEC_SINGLE_NODE_ROWS_THRESHOLD=<varname>number_of_rows</varname></codeblock>
+
+    <p>
+      <b>Type:</b> numeric
+    </p>
+
+    <p>
+      <b>Default:</b> 100
+    </p>
+
+    <p>
+      <b>Usage notes:</b> Typically, you increase the default value to make this optimization apply to more queries.
+      If incorrect or corrupted table and column statistics cause Impala to apply this optimization
+      incorrectly to queries that actually involve substantial work, you might see the queries being slower as a
+      result of remote reads. In that case, recompute statistics with the <codeph>COMPUTE STATS</codeph>
+      or <codeph>COMPUTE INCREMENTAL STATS</codeph> statement. If there is a problem collecting accurate
+      statistics, you can turn this feature off by setting the value to -1.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/internals_blurb"/>
+
+    <p>
+      This setting applies to query fragments where the amount of data to scan can be accurately determined, either
+      through table and column statistics, or by the presence of a <codeph>LIMIT</codeph> clause. If Impala cannot
+      accurately estimate the size of the input data, this setting does not apply.
+    </p>
+
+    <p rev="2.3.0">
+      In <keyword keyref="impala23_full"/> and higher, where Impala supports the complex data types <codeph>STRUCT</codeph>,
+      <codeph>ARRAY</codeph>, and <codeph>MAP</codeph>, if a query refers to any column of those types,
+      the small-query optimization is turned off for that query regardless of the
+      <codeph>EXEC_SINGLE_NODE_ROWS_THRESHOLD</codeph> setting.
+    </p>
+
+    <p>
+      For a query that is determined to be <q>small</q>, all work is performed on the coordinator node. This might
+      result in some I/O being performed by remote reads. The savings from not distributing the query work and not
+      generating native code are expected to outweigh any overhead from the remote reads.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/added_in_210"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      A common use case is to query just a few rows from a table to inspect typical data values. In this example,
+      Impala does not parallelize the query or perform native code generation because the result set is guaranteed
+      to be smaller than the threshold value from this query option:
+    </p>
+
+<codeblock>SET EXEC_SINGLE_NODE_ROWS_THRESHOLD=500;
+SELECT * FROM enormous_table LIMIT 300;
+</codeblock>
+
+<!-- Don't have any other places that tie into this particular optimization technique yet.
+Potentially: conceptual topics about code generation, distributed queries
+
+<p conref="../shared/impala_common.xml#common/related_info"/>
+<p>
+</p>
+-->
+
+  </conbody>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_explain.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_explain.xml b/docs/topics/impala_explain.xml
new file mode 100644
index 0000000..81cc17b
--- /dev/null
+++ b/docs/topics/impala_explain.xml
@@ -0,0 +1,228 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="explain">
+
+  <title>EXPLAIN Statement</title>
+  <titlealts audience="PDF"><navtitle>EXPLAIN</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Reports"/>
+      <data name="Category" value="Planning"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">EXPLAIN statement</indexterm>
+      Returns the execution plan for a statement, showing the low-level mechanisms that Impala will use to read the
+      data, divide the work among nodes in the cluster, and transmit intermediate and final results across the
+      network. Use <codeph>explain</codeph> followed by a complete <codeph>SELECT</codeph> query. For example:
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>EXPLAIN { <varname>select_query</varname> | <varname>ctas_stmt</varname> | <varname>insert_stmt</varname> }
+</codeblock>
+
+    <p>
+      The <varname>select_query</varname> is a <codeph>SELECT</codeph> statement, optionally prefixed by a
+      <codeph>WITH</codeph> clause. See <xref href="impala_select.xml#select"/> for details.
+    </p>
+
+    <p>
+      The <varname>insert_stmt</varname> is an <codeph>INSERT</codeph> statement that inserts into or overwrites an
+      existing table. It can use either the <codeph>INSERT ... SELECT</codeph> or <codeph>INSERT ...
+      VALUES</codeph> syntax. See <xref href="impala_insert.xml#insert"/> for details.
+    </p>
+
+    <p>
+      The <varname>ctas_stmt</varname> is a <codeph>CREATE TABLE</codeph> statement using the <codeph>AS
+      SELECT</codeph> clause, typically abbreviated as a <q>CTAS</q> operation. See
+      <xref href="impala_create_table.xml#create_table"/> for details.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      You can interpret the output to judge whether the query is performing efficiently, and adjust the query
+      and/or the schema if not. For example, you might change the tests in the <codeph>WHERE</codeph> clause, add
+      hints to make join operations more efficient, introduce subqueries, change the order of tables in a join, add
+      or change partitioning for a table, collect column statistics and/or table statistics in Hive, or any other
+      performance tuning steps.
+    </p>
+
+    <p>
+      The <codeph>EXPLAIN</codeph> output reminds you if table or column statistics are missing from any table
+      involved in the query. These statistics are important for optimizing queries involving large tables or
+      multi-table joins. See <xref href="impala_compute_stats.xml#compute_stats"/> for how to gather statistics,
+      and <xref href="impala_perf_stats.xml#perf_stats"/> for how to use this information for query tuning.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/explain_interpret"/>
+
+    <p>
+      If you come from a traditional database background and are not familiar with data warehousing, keep in mind
+      that Impala is optimized for full table scans across very large tables. The structure and distribution of
+      this data is typically not suitable for the kind of indexing and single-row lookups that are common in OLTP
+      environments. Seeing a query scan entirely through a large table is common, not necessarily an indication of
+      an inefficient query. Of course, if you can reduce the volume of scanned data by orders of magnitude, for
+      example by using a query that affects only certain partitions within a partitioned table, then you might be
+      able to optimize a query so that it executes in seconds rather than minutes.
+    </p>
+
+    <p>
+      For more information and examples to help you interpret <codeph>EXPLAIN</codeph> output, see
+      <xref href="impala_explain_plan.xml#perf_explain"/>.
+    </p>
+
+    <p rev="1.2">
+      <b>Extended EXPLAIN output:</b>
+    </p>
+
+    <p rev="1.2">
+      For performance tuning of complex queries, and capacity planning (such as using the admission control and
+      resource management features), you can enable more detailed and informative output for the
+      <codeph>EXPLAIN</codeph> statement. In the <cmdname>impala-shell</cmdname> interpreter, issue the command
+      <codeph>SET EXPLAIN_LEVEL=<varname>level</varname></codeph>, where <varname>level</varname> is an integer
+      from 0 to 3 or corresponding mnemonic values <codeph>minimal</codeph>, <codeph>standard</codeph>,
+      <codeph>extended</codeph>, or <codeph>verbose</codeph>.
+    </p>
+
+    <p rev="1.2">
+      When extended <codeph>EXPLAIN</codeph> output is enabled, <codeph>EXPLAIN</codeph> statements print
+      information about estimated memory requirements, minimum number of virtual cores, and so on.
+      <!--
+      that you can use to fine-tune the resource management options explained in <xref href="impala_resource_management.xml#rm_options"/>.
+      (The estimated memory requirements are intentionally on the high side, to allow a margin for error,
+      to avoid cancelling a query unnecessarily if you set the <codeph>MEM_LIMIT</codeph> option to the estimated memory figure.)
+      -->
+    </p>
+
+    <p>
+      See <xref href="impala_explain_level.xml#explain_level"/> for details and examples.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      This example shows how the standard <codeph>EXPLAIN</codeph> output moves from the lowest (physical) level to
+      the higher (logical) levels. The query begins by scanning a certain amount of data; each node performs an
+      aggregation operation (evaluating <codeph>COUNT(*)</codeph>) on some subset of data that is local to that
+      node; the intermediate results are transmitted back to the coordinator node (labelled here as the
+      <codeph>EXCHANGE</codeph> node); lastly, the intermediate results are summed to display the final result.
+    </p>
+
+<codeblock id="explain_plan_simple">[impalad-host:21000] &gt; explain select count(*) from customer_address;
++----------------------------------------------------------+
+| Explain String                                           |
++----------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=42.00MB VCores=1 |
+|                                                          |
+| 03:AGGREGATE [MERGE FINALIZE]                            |
+| |  output: sum(count(*))                                 |
+| |                                                        |
+| 02:EXCHANGE [PARTITION=UNPARTITIONED]                    |
+| |                                                        |
+| 01:AGGREGATE                                             |
+| |  output: count(*)                                      |
+| |                                                        |
+| 00:SCAN HDFS [default.customer_address]                  |
+|    partitions=1/1 size=5.25MB                            |
++----------------------------------------------------------+
+</codeblock>
+
+    <p>
+      These examples show how the extended <codeph>EXPLAIN</codeph> output becomes more accurate and informative as
+      statistics are gathered by the <codeph>COMPUTE STATS</codeph> statement. Initially, much of the information
+      about data size and distribution is marked <q>unavailable</q>. Impala can determine the raw data size, but
+      not the number of rows or number of distinct values for each column without additional analysis. The
+      <codeph>COMPUTE STATS</codeph> statement performs this analysis, so a subsequent <codeph>EXPLAIN</codeph>
+      statement has additional information to use in deciding how to optimize the distributed query.
+    </p>
+
+    <!-- To do:
+      Re-run these examples with more substantial tables populated with data.
+    -->
+
+<codeblock rev="1.2">[localhost:21000] &gt; set explain_level=extended;
+EXPLAIN_LEVEL set to extended
+[localhost:21000] &gt; explain select x from t1;
+[localhost:21000] &gt; explain select x from t1;
++----------------------------------------------------------+
+| Explain String                                           |
++----------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=32.00MB VCores=1 |
+|                                                          |
+| 01:EXCHANGE [PARTITION=UNPARTITIONED]                    |
+| |  hosts=1 per-host-mem=unavailable                      |
+<b>| |  tuple-ids=0 row-size=4B cardinality=unavailable       |</b>
+| |                                                        |
+| 00:SCAN HDFS [default.t2, PARTITION=RANDOM]              |
+|    partitions=1/1 size=36B                               |
+<b>|    table stats: unavailable                              |</b>
+<b>|    column stats: unavailable                             |</b>
+|    hosts=1 per-host-mem=32.00MB                          |
+<b>|    tuple-ids=0 row-size=4B cardinality=unavailable       |</b>
++----------------------------------------------------------+
+</codeblock>
+
+<codeblock rev="1.2">[localhost:21000] &gt; compute stats t1;
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 1 partition(s) and 1 column(s). |
++-----------------------------------------+
+[localhost:21000] &gt; explain select x from t1;
++----------------------------------------------------------+
+| Explain String                                           |
++----------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=64.00MB VCores=1 |
+|                                                          |
+| 01:EXCHANGE [PARTITION=UNPARTITIONED]                    |
+| |  hosts=1 per-host-mem=unavailable                      |
+| |  tuple-ids=0 row-size=4B cardinality=0                 |
+| |                                                        |
+| 00:SCAN HDFS [default.t1, PARTITION=RANDOM]              |
+|    partitions=1/1 size=36B                               |
+<b>|    table stats: 0 rows total                             |</b>
+<b>|    column stats: all                                     |</b>
+|    hosts=1 per-host-mem=64.00MB                          |
+<b>|    tuple-ids=0 row-size=4B cardinality=0                 |</b>
++----------------------------------------------------------+
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/security_blurb"/>
+    <p conref="../shared/impala_common.xml#common/redaction_yes"/>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+    <p rev="CDH-19187">
+      <!-- Doublecheck these details. Does EXPLAIN really need any permissions? -->
+      The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+      typically the <codeph>impala</codeph> user, must have read
+      and execute permissions for all applicable directories in all source tables
+      for the query that is being explained.
+      (A <codeph>SELECT</codeph> operation could read files from multiple different HDFS directories
+      if the source table is partitioned.)
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_select.xml#select"/>,
+      <xref href="impala_insert.xml#insert"/>,
+      <xref href="impala_create_table.xml#create_table"/>,
+      <xref href="impala_explain_plan.xml#explain_plan"/>
+    </p>
+
+  </conbody>
+</concept>


[02/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_txtfile.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_txtfile.xml b/docs/topics/impala_txtfile.xml
new file mode 100644
index 0000000..fc8f846
--- /dev/null
+++ b/docs/topics/impala_txtfile.xml
@@ -0,0 +1,812 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="txtfile">
+
+  <title>Using Text Data Files with Impala Tables</title>
+  <titlealts audience="PDF"><navtitle>Text Data Files</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="File Formats"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">Text support in Impala</indexterm>
+      Impala supports using text files as the storage format for input and output. Text files are a
+      convenient format to use for interchange with other applications or scripts that produce or read delimited
+      text files, such as CSV or TSV with commas or tabs for delimiters.
+    </p>
+
+    <p>
+      Text files are also very flexible in their column definitions. For example, a text file could have more
+      fields than the Impala table, and those extra fields are ignored during queries; or it could have fewer
+      fields than the Impala table, and those missing fields are treated as <codeph>NULL</codeph> values in
+      queries. You could have fields that were treated as numbers or timestamps in a table, then use <codeph>ALTER
+      TABLE ... REPLACE COLUMNS</codeph> to switch them to strings, or the reverse.
+    </p>
+
+    <table>
+      <title>Text Format Support in Impala</title>
+      <tgroup cols="5">
+        <colspec colname="1" colwidth="10*"/>
+        <colspec colname="2" colwidth="10*"/>
+        <colspec colname="3" colwidth="20*"/>
+        <colspec colname="4" colwidth="30*"/>
+        <colspec colname="5" colwidth="30*"/>
+        <thead>
+          <row>
+            <entry>
+              File Type
+            </entry>
+            <entry>
+              Format
+            </entry>
+            <entry>
+              Compression Codecs
+            </entry>
+            <entry>
+              Impala Can CREATE?
+            </entry>
+            <entry>
+              Impala Can INSERT?
+            </entry>
+          </row>
+        </thead>
+        <tbody>
+          <row conref="impala_file_formats.xml#file_formats/txtfile_support">
+            <entry/>
+          </row>
+        </tbody>
+      </tgroup>
+    </table>
+
+    <p outputclass="toc inpage"/>
+
+  </conbody>
+
+  <concept id="text_performance">
+
+    <title>Query Performance for Impala Text Tables</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Performance"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        Data stored in text format is relatively bulky, and not as efficient to query as binary formats such as
+        Parquet. You typically use text tables with Impala if that is the format you receive the data and you do
+        not have control over that process, or if you are a relatively new Hadoop user and not familiar with
+        techniques to generate files in other formats. (Because the default format for <codeph>CREATE
+        TABLE</codeph> is text, you might create your first Impala tables as text without giving performance much
+        thought.) Either way, look for opportunities to use more efficient file formats for the tables used in your
+        most performance-critical queries.
+      </p>
+
+      <p>
+        For frequently queried data, you might load the original text data files into one Impala table, then use an
+        <codeph>INSERT</codeph> statement to transfer the data to another table that uses the Parquet file format;
+        the data is converted automatically as it is stored in the destination table.
+      </p>
+
+      <p>
+        For more compact data, consider using LZO compression for the text files. LZO is the only compression codec
+        that Impala supports for text data, because the <q>splittable</q> nature of LZO data files lets different
+        nodes work on different parts of the same file in parallel. See <xref href="impala_txtfile.xml#lzo"/> for
+        details.
+      </p>
+
+      <p rev="2.0.0">
+        In Impala 2.0 and later, you can also use text data compressed in the gzip, bzip2, or Snappy formats.
+        Because these compressed formats are not <q>splittable</q> in the way that LZO is, there is less
+        opportunity for Impala to parallelize queries on them. Therefore, use these types of compressed data only
+        for convenience if that is the format in which you receive the data. Prefer to use LZO compression for text
+        data if you have the choice, or convert the data to Parquet using an <codeph>INSERT ... SELECT</codeph>
+        statement to copy the original data into a Parquet table.
+      </p>
+
+      <note rev="2.2.0">
+        <p>
+          Impala supports bzip files created by the <codeph>bzip2</codeph> command, but not bzip files with
+          multiple streams created by the <codeph>pbzip2</codeph> command. Impala decodes only the data from the
+          first part of such files, leading to incomplete results.
+        </p>
+
+        <p>
+          The maximum size that Impala can accommodate for an individual bzip file is 1 GB (after uncompression).
+        </p>
+      </note>
+
+      <p conref="../shared/impala_common.xml#common/s3_block_splitting"/>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="text_ddl">
+
+    <title>Creating Text Tables</title>
+
+    <conbody>
+
+      <p>
+        <b>To create a table using text data files:</b>
+      </p>
+
+      <p>
+        If the exact format of the text data files (such as the delimiter character) is not significant, use the
+        <codeph>CREATE TABLE</codeph> statement with no extra clauses at the end to create a text-format table. For
+        example:
+      </p>
+
+<codeblock>create table my_table(id int, s string, n int, t timestamp, b boolean);
+</codeblock>
+
+      <p>
+        The data files created by any <codeph>INSERT</codeph> statements will use the Ctrl-A character (hex 01) as
+        a separator between each column value.
+      </p>
+
+      <p>
+        A common use case is to import existing text files into an Impala table. The syntax is more verbose; the
+        significant part is the <codeph>FIELDS TERMINATED BY</codeph> clause, which must be preceded by the
+        <codeph>ROW FORMAT DELIMITED</codeph> clause. The statement can end with a <codeph>STORED AS
+        TEXTFILE</codeph> clause, but that clause is optional because text format tables are the default. For
+        example:
+      </p>
+
+<codeblock>create table csv(id int, s string, n int, t timestamp, b boolean)
+  row format delimited
+  <ph id="csv">fields terminated by ',';</ph>
+
+create table tsv(id int, s string, n int, t timestamp, b boolean)
+  row format delimited
+  <ph id="tsv">fields terminated by '\t';</ph>
+
+create table pipe_separated(id int, s string, n int, t timestamp, b boolean)
+  row format delimited
+  <ph id="psv">fields terminated by '|'</ph>
+  stored as textfile;
+</codeblock>
+
+      <p>
+        You can create tables with specific separator characters to import text files in familiar formats such as
+        CSV, TSV, or pipe-separated. You can also use these tables to produce output data files, by copying data
+        into them through the <codeph>INSERT ... SELECT</codeph> syntax and then extracting the data files from the
+        Impala data directory.
+      </p>
+
+      <p rev="1.3.1">
+        In Impala 1.3.1 and higher, you can specify a delimiter character <codeph>'\</codeph><codeph>0'</codeph> to
+        use the ASCII 0 (<codeph>nul</codeph>) character for text tables:
+      </p>
+
+<codeblock rev="1.3.1">create table nul_separated(id int, s string, n int, t timestamp, b boolean)
+  row format delimited
+  fields terminated by '\0'
+  stored as textfile;
+</codeblock>
+
+      <note>
+        <p>
+          Do not surround string values with quotation marks in text data files that you construct. If you need to
+          include the separator character inside a field value, for example to put a string value with a comma
+          inside a CSV-format data file, specify an escape character on the <codeph>CREATE TABLE</codeph> statement
+          with the <codeph>ESCAPED BY</codeph> clause, and insert that character immediately before any separator
+          characters that need escaping.
+        </p>
+      </note>
+
+<!--
+      <p>
+        In the <cmdname>impala-shell</cmdname> interpreter, issue a command similar to:
+      </p>
+
+<codeblock>create table textfile_table (<varname>column_specs</varname>) stored as textfile;
+/* If the STORED AS clause is omitted, the default is a TEXTFILE with hex 01 characters as the delimiter. */
+create table default_table (<varname>column_specs</varname>);
+/* Some optional clauses in the CREATE TABLE statement apply only to Text tables. */
+create table csv_table (<varname>column_specs</varname>) row format delimited fields terminated by ',';
+create table tsv_table (<varname>column_specs</varname>) row format delimited fields terminated by '\t';
+create table dos_table (<varname>column_specs</varname>) lines terminated by '\r';</codeblock>
+-->
+
+      <p>
+        Issue a <codeph>DESCRIBE FORMATTED <varname>table_name</varname></codeph> statement to see the details of
+        how each table is represented internally in Impala.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_unsupported_filetype"/>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="text_data_files">
+
+    <title>Data Files for Text Tables</title>
+
+    <conbody>
+
+      <p>
+        When Impala queries a table with data in text format, it consults all the data files in the data directory
+        for that table, with some exceptions:
+      </p>
+
+      <ul rev="2.2.0">
+        <li>
+          <p>
+            Impala ignores any hidden files, that is, files whose names start with a dot or an underscore.
+          </p>
+        </li>
+
+        <li>
+          <p conref="../shared/impala_common.xml#common/ignore_file_extensions"/>
+        </li>
+
+        <li>
+<!-- Copied and slightly adapted text from later on in this same file. Turn into a conref. -->
+          <p>
+            Impala uses suffixes to recognize when text data files are compressed text. For Impala to recognize the
+            compressed text files, they must have the appropriate file extension corresponding to the compression
+            codec, either <codeph>.gz</codeph>, <codeph>.bz2</codeph>, or <codeph>.snappy</codeph>. The extensions
+            can be in uppercase or lowercase.
+          </p>
+        </li>
+
+        <li>
+          Otherwise, the file names are not significant. When you put files into an HDFS directory through ETL
+          jobs, or point Impala to an existing HDFS directory with the <codeph>CREATE EXTERNAL TABLE</codeph>
+          statement, or move data files under external control with the <codeph>LOAD DATA</codeph> statement,
+          Impala preserves the original filenames.
+        </li>
+      </ul>
+
+      <p>
+        Filenames for data produced through Impala <codeph>INSERT</codeph> statements are given unique names to
+        avoid filename conflicts.
+      </p>
+
+      <p>
+        An <codeph>INSERT ... SELECT</codeph> statement produces one data file from each node that processes the
+        <codeph>SELECT</codeph> part of the statement. An <codeph>INSERT ... VALUES</codeph> statement produces a
+        separate data file for each statement; because Impala is more efficient querying a small number of huge
+        files than a large number of tiny files, the <codeph>INSERT ... VALUES</codeph> syntax is not recommended
+        for loading a substantial volume of data. If you find yourself with a table that is inefficient due to too
+        many small data files, reorganize the data into a few large files by doing <codeph>INSERT ...
+        SELECT</codeph> to transfer the data to a new table.
+      </p>
+
+      <p>
+        <b>Special values within text data files:</b>
+      </p>
+
+      <ul>
+        <li rev="1.4.0">
+          <p>
+            Impala recognizes the literal strings <codeph>inf</codeph> for infinity and <codeph>nan</codeph> for
+            <q>Not a Number</q>, for <codeph>FLOAT</codeph> and <codeph>DOUBLE</codeph> columns.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Impala recognizes the literal string <codeph>\N</codeph> to represent <codeph>NULL</codeph>. When using
+            Sqoop, specify the options <codeph>--null-non-string</codeph> and <codeph>--null-string</codeph> to
+            ensure all <codeph>NULL</codeph> values are represented correctly in the Sqoop output files. By default,
+            Sqoop writes <codeph>NULL</codeph> values using the string <codeph>null</codeph>, which causes a
+            conversion error when such rows are evaluated by Impala. (A workaround for existing tables and data files
+            is to change the table properties through <codeph>ALTER TABLE <varname>name</varname> SET
+            TBLPROPERTIES("serialization.null.format"="null")</codeph>.)
+          </p>
+        </li>
+
+        <li>
+          <p conref="../shared/impala_common.xml#common/skip_header_lines"/>
+        </li>
+      </ul>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="text_etl">
+
+    <title>Loading Data into Impala Text Tables</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="ETL"/>
+      <data name="Category" value="Ingest"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        To load an existing text file into an Impala text table, use the <codeph>LOAD DATA</codeph> statement and
+        specify the path of the file in HDFS. That file is moved into the appropriate Impala data directory.
+      </p>
+
+      <p>
+        To load multiple existing text files into an Impala text table, use the <codeph>LOAD DATA</codeph>
+        statement and specify the HDFS path of the directory containing the files. All non-hidden files are moved
+        into the appropriate Impala data directory.
+      </p>
+
+      <p>
+        To convert data to text from any other file format supported by Impala, use a SQL statement such as:
+      </p>
+
+<codeblock>-- Text table with default delimiter, the hex 01 character.
+CREATE TABLE text_table AS SELECT * FROM other_file_format_table;
+
+-- Text table with user-specified delimiter. Currently, you cannot specify
+-- the delimiter as part of CREATE TABLE LIKE or CREATE TABLE AS SELECT.
+-- But you can change an existing text table to have a different delimiter.
+CREATE TABLE csv LIKE other_file_format_table;
+ALTER TABLE csv SET SERDEPROPERTIES ('serialization.format'=',', 'field.delim'=',');
+INSERT INTO csv SELECT * FROM other_file_format_table;</codeblock>
+
+      <p>
+        This can be a useful technique to see how Impala represents special values within a text-format data file.
+        Use the <codeph>DESCRIBE FORMATTED</codeph> statement to see the HDFS directory where the data files are
+        stored, then use Linux commands such as <codeph>hdfs dfs -ls <varname>hdfs_directory</varname></codeph> and
+        <codeph>hdfs dfs -cat <varname>hdfs_file</varname></codeph> to display the contents of an Impala-created
+        text file.
+      </p>
+
+      <p>
+        To create a few rows in a text table for test purposes, you can use the <codeph>INSERT ... VALUES</codeph>
+        syntax:
+      </p>
+
+<codeblock>INSERT INTO <varname>text_table</varname> VALUES ('string_literal',100,hex('hello world'));</codeblock>
+
+      <note>
+        Because Impala and the HDFS infrastructure are optimized for multi-megabyte files, avoid the <codeph>INSERT
+        ... VALUES</codeph> notation when you are inserting many rows. Each <codeph>INSERT ... VALUES</codeph>
+        statement produces a new tiny file, leading to fragmentation and reduced performance. When creating any
+        substantial volume of new data, use one of the bulk loading techniques such as <codeph>LOAD DATA</codeph>
+        or <codeph>INSERT ... SELECT</codeph>. Or, <xref href="impala_hbase.xml#impala_hbase">use an HBase
+        table</xref> for single-row <codeph>INSERT</codeph> operations, because HBase tables are not subject to the
+        same fragmentation issues as tables stored on HDFS.
+      </note>
+
+      <p>
+        When you create a text file for use with an Impala text table, specify <codeph>\N</codeph> to represent a
+        <codeph>NULL</codeph> value. For the differences between <codeph>NULL</codeph> and empty strings, see
+        <xref href="impala_literals.xml#null"/>.
+      </p>
+
+      <p>
+        If a text file has fewer fields than the columns in the corresponding Impala table, all the corresponding
+        columns are set to <codeph>NULL</codeph> when the data in that file is read by an Impala query.
+      </p>
+
+      <p>
+        If a text file has more fields than the columns in the corresponding Impala table, the extra fields are
+        ignored when the data in that file is read by an Impala query.
+      </p>
+
+      <p>
+        You can also use manual HDFS operations such as <codeph>hdfs dfs -put</codeph> or <codeph>hdfs dfs
+        -cp</codeph> to put data files in the data directory for an Impala table. When you copy or move new data
+        files into the HDFS directory for the Impala table, issue a <codeph>REFRESH
+        <varname>table_name</varname></codeph> statement in <cmdname>impala-shell</cmdname> before issuing the next
+        query against that table, to make Impala recognize the newly added files.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="lzo">
+
+    <title>Using LZO-Compressed Text Files</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="LZO"/>
+      <data name="Category" value="Compression"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">LZO support in Impala</indexterm>
+
+        <indexterm audience="Cloudera">compression</indexterm>
+        Impala supports using text data files that employ LZO compression. <ph rev="upstream">Cloudera</ph> recommends compressing
+        text data files when practical. Impala queries are usually I/O-bound; reducing the amount of data read from
+        disk typically speeds up a query, despite the extra CPU work to uncompress the data in memory.
+      </p>
+
+      <p>
+        Impala can work with LZO-compressed text files are preferable to files compressed by other codecs, because
+        LZO-compressed files are <q>splittable</q>, meaning that different portions of a file can be uncompressed
+        and processed independently by different nodes.
+      </p>
+
+      <p>
+        Impala does not currently support writing LZO-compressed text files.
+      </p>
+
+      <p>
+        Because Impala can query LZO-compressed files but currently cannot write them, you use Hive to do the
+        initial <codeph>CREATE TABLE</codeph> and load the data, then switch back to Impala to run queries. For
+        instructions on setting up LZO compression for Hive <codeph>CREATE TABLE</codeph> and
+        <codeph>INSERT</codeph> statements, see
+        <xref href="https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LZO" scope="external" format="html">the
+        LZO page on the Hive wiki</xref>. Once you have created an LZO text table, you can also manually add
+        LZO-compressed text files to it, produced by the
+        <xref href="http://www.lzop.org/" scope="external" format="html"> <cmdname>lzop</cmdname></xref> command
+        or similar method.
+      </p>
+
+      <section id="lzo_setup">
+
+        <title>Preparing to Use LZO-Compressed Text Files</title>
+
+        <p>
+          Before using LZO-compressed tables in Impala, do the following one-time setup for each machine in the
+          cluster. Install the necessary packages using either the Cloudera public repository, a private repository
+          you establish, or by using packages. You must do these steps manually, whether or not the cluster is
+          managed by the Cloudera Manager product.
+        </p>
+
+        <ol>
+          <li>
+            <b>Prepare your systems to work with LZO by downloading and installing the appropriate libraries:</b>
+            <p>
+              <b>On systems managed by Cloudera Manager using parcels:</b>
+            </p>
+
+            <p>
+              See the setup instructions for the LZO parcel in the Cloudera Manager documentation for
+              <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_gpl_extras.html" scope="external" format="html">Cloudera
+              Manager 5</xref>.
+            </p>
+
+            <p>
+              <b>On systems managed by Cloudera Manager using packages, or not managed by Cloudera Manager:</b>
+            </p>
+
+            <p>
+              Download and install the appropriate file to each machine on which you intend to use LZO with Impala.
+              These files all come from the Cloudera
+              <xref href="https://archive.cloudera.com/gplextras/redhat/5/x86_64/gplextras/" scope="external" format="html">GPL
+              extras</xref> download site. Install the:
+            </p>
+            <ul>
+              <li>
+                <xref href="https://archive.cloudera.com/gplextras/redhat/5/x86_64/gplextras/cloudera-gplextras4.repo" scope="external" format="repo">Red
+                Hat 5 repo file</xref> to <filepath>/etc/yum.repos.d/</filepath>.
+              </li>
+
+              <li>
+                <xref href="https://archive.cloudera.com/gplextras/redhat/6/x86_64/gplextras/cloudera-gplextras4.repo" scope="external" format="repo">Red
+                Hat 6 repo file</xref> to <filepath>/etc/yum.repos.d/</filepath>.
+              </li>
+
+              <li>
+                <xref href="https://archive.cloudera.com/gplextras/sles/11/x86_64/gplextras/cloudera-gplextras4.repo" scope="external" format="repo">SUSE
+                repo file</xref> to <filepath>/etc/zypp/repos.d/</filepath>.
+              </li>
+
+              <li>
+                <xref href="https://archive.cloudera.com/gplextras/ubuntu/lucid/amd64/gplextras/cloudera.list" scope="external" format="list">Ubuntu
+                10.04 list file</xref> to <filepath>/etc/apt/sources.list.d/</filepath>.
+              </li>
+
+              <li>
+                <xref href="https://archive.cloudera.com/gplextras/ubuntu/precise/amd64/gplextras/cloudera.list" scope="external" format="list">Ubuntu
+                12.04 list file</xref> to <filepath>/etc/apt/sources.list.d/</filepath>.
+              </li>
+
+              <li>
+                <xref href="https://archive.cloudera.com/gplextras/debian/squeeze/amd64/gplextras/cloudera.list" scope="external" format="list">Debian
+                list file</xref> to <filepath>/etc/apt/sources.list.d/</filepath>.
+              </li>
+            </ul>
+          </li>
+
+          <li>
+            <b>Configure Impala to use LZO:</b>
+            <p>
+              Use <b>one</b> of the following sets of commands to refresh your package management system's
+              repository information, install the base LZO support for Hadoop, and install the LZO support for
+              Impala.
+            </p>
+
+            <note rev="1.2.0">
+              <p rev="1.2.0">
+                <ph rev="upstream">The name of the Hadoop LZO package changed between CDH 4 and CDH 5. In CDH 4, the package name was
+                <codeph>hadoop-lzo-cdh4</codeph>. In CDH 5 and higher, the package name is <codeph>hadoop-lzo</codeph>.</ph>
+              </p>
+            </note>
+
+            <p>
+              <b>For RHEL/CentOS systems:</b>
+            </p>
+<codeblock>$ sudo yum update
+$ sudo yum install hadoop-lzo
+$ sudo yum install impala-lzo</codeblock>
+            <p>
+              <b>For SUSE systems:</b>
+            </p>
+<codeblock rev="1.2">$ sudo apt-get update
+$ sudo zypper install hadoop-lzo
+$ sudo zypper install impala-lzo</codeblock>
+            <p>
+              <b>For Debian/Ubuntu systems:</b>
+            </p>
+<codeblock>$ sudo zypper update
+$ sudo apt-get install hadoop-lzo
+$ sudo apt-get install impala-lzo</codeblock>
+            <note>
+              <p>
+                The level of the <codeph>impala-lzo-cdh4</codeph> package is closely tied to the version of Impala
+                you use. Any time you upgrade Impala, re-do the installation command for
+                <codeph>impala-lzo</codeph> on each applicable machine to make sure you have the appropriate
+                version of that package.
+              </p>
+            </note>
+          </li>
+
+          <li>
+            For <codeph>core-site.xml</codeph> on the client <b>and</b> server (that is, in the configuration
+            directories for both Impala and Hadoop), append <codeph>com.hadoop.compression.lzo.LzopCodec</codeph>
+            to the comma-separated list of codecs. For example:
+<codeblock>&lt;property&gt;
+  &lt;name&gt;io.compression.codecs&lt;/name&gt;
+  &lt;value&gt;org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,
+        org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,
+        org.apache.hadoop.io.compress.SnappyCodec,com.hadoop.compression.lzo.LzopCodec&lt;/value&gt;
+&lt;/property&gt;</codeblock>
+            <note>
+              <p>
+                If this is the first time you have edited the Hadoop <filepath>core-site.xml</filepath> file, note
+                that the <filepath>/etc/hadoop/conf</filepath> directory is typically a symbolic link, so the
+                canonical <filepath>core-site.xml</filepath> might reside in a different directory:
+              </p>
+<codeblock>$ ls -l /etc/hadoop
+total 8
+lrwxrwxrwx. 1 root root   29 Feb 26  2013 conf -&gt; /etc/alternatives/hadoop-conf
+lrwxrwxrwx. 1 root root   10 Feb 26  2013 conf.dist -&gt; conf.empty
+drwxr-xr-x. 2 root root 4096 Feb 26  2013 conf.empty
+drwxr-xr-x. 2 root root 4096 Oct 28 15:46 conf.pseudo</codeblock>
+              <p>
+                If the <codeph>io.compression.codecs</codeph> property is missing from
+                <filepath>core-site.xml</filepath>, only add <codeph>com.hadoop.compression.lzo.LzopCodec</codeph>
+                to the new property value, not all the names from the preceding example.
+              </p>
+            </note>
+          </li>
+
+          <li>
+            <!-- To do:
+              Link to CM or other doc where that procedure is explained.
+              Run through the procedure in CM and cite the relevant safety valves to put the XML into.
+            -->
+            Restart the MapReduce and Impala services.
+          </li>
+        </ol>
+
+      </section>
+
+      <section id="lzo_create_table">
+
+        <title>Creating LZO Compressed Text Tables</title>
+
+        <p>
+          A table containing LZO-compressed text files must be created in Hive with the following storage clause:
+        </p>
+
+<codeblock>STORED AS
+    INPUTFORMAT 'com.hadoop.mapred.DeprecatedLzoTextInputFormat'
+    OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'</codeblock>
+
+<!--
+      <p>
+        In Hive, when writing LZO compressed text tables, you must include the following specification:
+      </p>
+
+<codeblock>hive&gt; SET hive.exec.compress.output=true;
+hive&gt; SET mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec;</codeblock>
+-->
+
+        <p>
+          Also, certain Hive settings need to be in effect. For example:
+        </p>
+
+<codeblock>hive&gt; SET mapreduce.output.fileoutputformat.compress=true;
+hive&gt; SET hive.exec.compress.output=true;
+hive&gt; SET mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec;
+hive&gt; CREATE TABLE lzo_t (s string) STORED AS
+  &gt; INPUTFORMAT 'com.hadoop.mapred.DeprecatedLzoTextInputFormat'
+  &gt; OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';
+hive&gt; INSERT INTO TABLE lzo_t SELECT col1, col2 FROM uncompressed_text_table;</codeblock>
+
+        <p>
+          Once you have created LZO-compressed text tables, you can convert data stored in other tables (regardless
+          of file format) by using the <codeph>INSERT ... SELECT</codeph> statement in Hive.
+        </p>
+
+        <p>
+          Files in an LZO-compressed table must use the <codeph>.lzo</codeph> extension. Examine the files in the
+          HDFS data directory after doing the <codeph>INSERT</codeph> in Hive, to make sure the files have the
+          right extension. If the required settings are not in place, you end up with regular uncompressed files,
+          and Impala cannot access the table because it finds data files with the wrong (uncompressed) format.
+        </p>
+
+        <p>
+          After loading data into an LZO-compressed text table, index the files so that they can be split. You
+          index the files by running a Java class,
+          <codeph>com.hadoop.compression.lzo.DistributedLzoIndexer</codeph>, through the Linux command line. This
+          Java class is included in the <codeph>hadoop-lzo</codeph> package.
+        </p>
+
+        <p>
+          Run the indexer using a command like the following:
+        </p>
+
+<codeblock>$ hadoop jar /usr/lib/hadoop/lib/hadoop-lzo-cdh4-0.4.15-gplextras.jar
+  com.hadoop.compression.lzo.DistributedLzoIndexer /hdfs_location_of_table/</codeblock>
+
+        <note>
+          If the path of the JAR file in the preceding example is not recognized, do a <cmdname>find</cmdname>
+          command to locate <filepath>hadoop-lzo-*-gplextras.jar</filepath> and use that path.
+        </note>
+
+        <p>
+          Indexed files have the same name as the file they index, with the <codeph>.index</codeph> extension. If
+          the data files are not indexed, Impala queries still work, but the queries read the data from remote
+          DataNodes, which is very inefficient.
+        </p>
+
+        <!-- To do:
+          Here is the place to put some end-to-end examples once I have it
+          all working. Or at least the final step with Impala queries.
+          Have never actually gotten this part working yet due to mismatches
+          between the levels of Impala and LZO packages.
+        -->
+
+        <p>
+          Once the LZO-compressed tables are created, and data is loaded and indexed, you can query them through
+          Impala. As always, the first time you start <cmdname>impala-shell</cmdname> after creating a table in
+          Hive, issue an <codeph>INVALIDATE METADATA</codeph> statement so that Impala recognizes the new table.
+          (In Impala 1.2 and higher, you only have to run <codeph>INVALIDATE METADATA</codeph> on one node, rather
+          than on all the Impala nodes.)
+        </p>
+
+      </section>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.0.0" id="gzip">
+
+    <title>Using gzip, bzip2, or Snappy-Compressed Text Files</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Snappy"/>
+      <data name="Category" value="Gzip"/>
+      <data name="Category" value="Compression"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">gzip support in Impala</indexterm>
+
+        <indexterm audience="Cloudera">bzip2 support in Impala</indexterm>
+
+        <indexterm audience="Cloudera">Snappy support in Impala</indexterm>
+
+        <indexterm audience="Cloudera">compression</indexterm>
+        In Impala 2.0 and later, Impala supports using text data files that employ gzip, bzip2, or Snappy
+        compression. These compression types are primarily for convenience within an existing ETL pipeline rather
+        than maximum performance. Although it requires less I/O to read compressed text than the equivalent
+        uncompressed text, files compressed by these codecs are not <q>splittable</q> and therefore cannot take
+        full advantage of the Impala parallel query capability.
+      </p>
+
+      <p>
+        As each bzip2- or Snappy-compressed text file is processed, the node doing the work reads the entire file
+        into memory and then decompresses it. Therefore, the node must have enough memory to hold both the
+        compressed and uncompressed data from the text file. The memory required to hold the uncompressed data is
+        difficult to estimate in advance, potentially causing problems on systems with low memory limits or with
+        resource management enabled. <ph rev="2.1.0">In Impala 2.1 and higher, this memory overhead is reduced for
+        gzip-compressed text files. The gzipped data is decompressed as it is read, rather than all at once.</ph>
+      </p>
+
+<!--
+    <p>
+    Impala can work with LZO-compressed text files but not GZip-compressed text.
+    LZO-compressed files are <q>splittable</q>, meaning that different portions of a file
+    can be uncompressed and processed independently by different nodes. GZip-compressed
+    files are not splittable, making them unsuitable for Impala-style distributed queries.
+    </p>
+-->
+
+      <p>
+        To create a table to hold gzip, bzip2, or Snappy-compressed text, create a text table with no special
+        compression options. Specify the delimiter and escape character if required, using the <codeph>ROW
+        FORMAT</codeph> clause.
+      </p>
+
+      <p>
+        Because Impala can query compressed text files but currently cannot write them, produce the compressed text
+        files outside Impala and use the <codeph>LOAD DATA</codeph> statement, manual HDFS commands to move them to
+        the appropriate Impala data directory. (Or, you can use <codeph>CREATE EXTERNAL TABLE</codeph> and point
+        the <codeph>LOCATION</codeph> attribute at a directory containing existing compressed text files.)
+      </p>
+
+      <p>
+        For Impala to recognize the compressed text files, they must have the appropriate file extension
+        corresponding to the compression codec, either <codeph>.gz</codeph>, <codeph>.bz2</codeph>, or
+        <codeph>.snappy</codeph>. The extensions can be in uppercase or lowercase.
+      </p>
+
+      <p>
+        The following example shows how you can create a regular text table, put different kinds of compressed and
+        uncompressed files into it, and Impala automatically recognizes and decompresses each one based on their
+        file extensions:
+      </p>
+
+<codeblock>create table csv_compressed (a string, b string, c string)
+  row format delimited fields terminated by ",";
+
+insert into csv_compressed values
+  ('one - uncompressed', 'two - uncompressed', 'three - uncompressed'),
+  ('abc - uncompressed', 'xyz - uncompressed', '123 - uncompressed');
+...make equivalent .gz, .bz2, and .snappy files and load them into same table directory...
+
+select * from csv_compressed;
++--------------------+--------------------+----------------------+
+| a                  | b                  | c                    |
++--------------------+--------------------+----------------------+
+| one - snappy       | two - snappy       | three - snappy       |
+| one - uncompressed | two - uncompressed | three - uncompressed |
+| abc - uncompressed | xyz - uncompressed | 123 - uncompressed   |
+| one - bz2          | two - bz2          | three - bz2          |
+| abc - bz2          | xyz - bz2          | 123 - bz2            |
+| one - gzip         | two - gzip         | three - gzip         |
+| abc - gzip         | xyz - gzip         | 123 - gzip           |
++--------------------+--------------------+----------------------+
+
+$ hdfs dfs -ls 'hdfs://127.0.0.1:8020/user/hive/warehouse/file_formats.db/csv_compressed/';
+...truncated for readability...
+75 hdfs://127.0.0.1:8020/user/hive/warehouse/file_formats.db/csv_compressed/csv_compressed.snappy
+79 hdfs://127.0.0.1:8020/user/hive/warehouse/file_formats.db/csv_compressed/csv_compressed_bz2.csv.bz2
+80 hdfs://127.0.0.1:8020/user/hive/warehouse/file_formats.db/csv_compressed/csv_compressed_gzip.csv.gz
+116 hdfs://127.0.0.1:8020/user/hive/warehouse/file_formats.db/csv_compressed/dd414df64d67d49b_data.0.
+</codeblock>
+
+    </conbody>
+
+  </concept>
+
+  <concept audience="Cloudera" id="txtfile_data_types">
+
+    <title>Data Type Considerations for Text Tables</title>
+
+    <conbody>
+
+      <p></p>
+
+    </conbody>
+
+  </concept>
+
+</concept>


[12/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_porting.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_porting.xml b/docs/topics/impala_porting.xml
new file mode 100644
index 0000000..3800713
--- /dev/null
+++ b/docs/topics/impala_porting.xml
@@ -0,0 +1,623 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="porting">
+
+  <title>Porting SQL from Other Database Systems to Impala</title>
+  <titlealts audience="PDF"><navtitle>Porting SQL</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Databases"/>
+      <data name="Category" value="Hive"/>
+      <data name="Category" value="Oracle"/>
+      <data name="Category" value="MySQL"/>
+      <data name="Category" value="PostgreSQL"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Porting"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">porting</indexterm>
+      Although Impala uses standard SQL for queries, you might need to modify SQL source when bringing applications
+      to Impala, due to variations in data types, built-in functions, vendor language extensions, and
+      Hadoop-specific syntax. Even when SQL is working correctly, you might make further minor modifications for
+      best performance.
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="porting_ddl_dml">
+
+    <title>Porting DDL and DML Statements</title>
+
+    <conbody>
+
+      <p>
+        When adapting SQL code from a traditional database system to Impala, expect to find a number of differences
+        in the DDL statements that you use to set up the schema. Clauses related to physical layout of files,
+        tablespaces, and indexes have no equivalent in Impala. You might restructure your schema considerably to
+        account for the Impala partitioning scheme and Hadoop file formats.
+      </p>
+
+      <p>
+        Expect SQL queries to have a much higher degree of compatibility. With modest rewriting to address vendor
+        extensions and features not yet supported in Impala, you might be able to run identical or almost-identical
+        query text on both systems.
+      </p>
+
+      <p>
+        Therefore, consider separating out the DDL into a separate Impala-specific setup script. Focus your reuse
+        and ongoing tuning efforts on the code for SQL queries.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="porting_data_types">
+
+    <title>Porting Data Types from Other Database Systems</title>
+
+    <conbody>
+
+      <ul>
+        <li>
+          <p>
+            Change any <codeph>VARCHAR</codeph>, <codeph>VARCHAR2</codeph>, and <codeph>CHAR</codeph> columns to
+            <codeph>STRING</codeph>. Remove any length constraints from the column declarations; for example,
+            change <codeph>VARCHAR(32)</codeph> or <codeph>CHAR(1)</codeph> to <codeph>STRING</codeph>. Impala is
+            very flexible about the length of string values; it does not impose any length constraints
+            or do any special processing (such as blank-padding) for <codeph>STRING</codeph> columns.
+            (In Impala 2.0 and higher, there are data types <codeph>VARCHAR</codeph> and <codeph>CHAR</codeph>,
+            with length constraints for both types and blank-padding for <codeph>CHAR</codeph>.
+            However, for performance reasons, it is still preferable to use <codeph>STRING</codeph>
+            columns where practical.)
+          </p>
+        </li>
+
+        <li>
+          <p>
+            For national language character types such as <codeph>NCHAR</codeph>, <codeph>NVARCHAR</codeph>, or
+            <codeph>NCLOB</codeph>, be aware that while Impala can store and query UTF-8 character data, currently
+            some string manipulation operations only work correctly with ASCII data. See
+            <xref href="impala_string.xml#string"/> for details.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Change any <codeph>DATE</codeph>, <codeph>DATETIME</codeph>, or <codeph>TIME</codeph> columns to
+            <codeph>TIMESTAMP</codeph>. Remove any precision constraints. Remove any timezone clauses, and make
+            sure your application logic or ETL process accounts for the fact that Impala expects all
+            <codeph>TIMESTAMP</codeph> values to be in
+            <xref href="http://en.wikipedia.org/wiki/Coordinated_Universal_Time" scope="external" format="html">Coordinated
+            Universal Time (UTC)</xref>. See <xref href="impala_timestamp.xml#timestamp"/> for information about
+            the <codeph>TIMESTAMP</codeph> data type, and
+            <xref href="impala_datetime_functions.xml#datetime_functions"/> for conversion functions for different
+            date and time formats.
+          </p>
+          <p>
+            You might also need to adapt date- and time-related literal values and format strings to use the
+            supported Impala date and time formats. If you have date and time literals with different separators or
+            different numbers of <codeph>YY</codeph>, <codeph>MM</codeph>, and so on placeholders than Impala
+            expects, consider using calls to <codeph>regexp_replace()</codeph> to transform those values to the
+            Impala-compatible format. See <xref href="impala_timestamp.xml#timestamp"/> for information about the
+            allowed formats for date and time literals, and
+            <xref href="impala_string_functions.xml#string_functions"/> for string conversion functions such as
+            <codeph>regexp_replace()</codeph>.
+          </p>
+          <p>
+            Instead of <codeph>SYSDATE</codeph>, call the function <codeph>NOW()</codeph>.
+          </p>
+          <p>
+            Instead of adding or subtracting directly from a date value to produce a value <varname>N</varname>
+            days in the past or future, use an <codeph>INTERVAL</codeph> expression, for example <codeph>NOW() +
+            INTERVAL 30 DAYS</codeph>.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Although Impala supports <codeph>INTERVAL</codeph> expressions for datetime arithmetic, as shown in
+            <xref href="impala_timestamp.xml#timestamp"/>, <codeph>INTERVAL</codeph> is not available as a column
+            data type in Impala. For any <codeph>INTERVAL</codeph> values stored in tables, convert them to numeric
+            values that you can add or subtract using the functions in
+            <xref href="impala_datetime_functions.xml#datetime_functions"/>. For example, if you had a table
+            <codeph>DEADLINES</codeph> with an <codeph>INT</codeph> column <codeph>TIME_PERIOD</codeph>, you could
+            construct dates N days in the future like so:
+          </p>
+<codeblock>SELECT NOW() + INTERVAL time_period DAYS from deadlines;</codeblock>
+        </li>
+
+        <li>
+          <p>
+            For <codeph>YEAR</codeph> columns, change to the smallest Impala integer type that has sufficient
+            range. See <xref href="impala_datatypes.xml#datatypes"/> for details about ranges, casting, and so on
+            for the various numeric data types.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Change any <codeph>DECIMAL</codeph> and <codeph>NUMBER</codeph> types. If fixed-point precision is not
+            required, you can use <codeph>FLOAT</codeph> or <codeph>DOUBLE</codeph> on the Impala side depending on
+            the range of values. For applications that require precise decimal values, such as financial data, you
+            might need to make more extensive changes to table structure and application logic, such as using
+            separate integer columns for dollars and cents, or encoding numbers as string values and writing UDFs
+            to manipulate them. See <xref href="impala_datatypes.xml#datatypes"/> for details about ranges,
+            casting, and so on for the various numeric data types.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            <codeph>FLOAT</codeph>, <codeph>DOUBLE</codeph>, and <codeph>REAL</codeph> types are supported in
+            Impala. Remove any precision and scale specifications. (In Impala, <codeph>REAL</codeph> is just an
+            alias for <codeph>DOUBLE</codeph>; columns declared as <codeph>REAL</codeph> are turned into
+            <codeph>DOUBLE</codeph> behind the scenes.) See <xref href="impala_datatypes.xml#datatypes"/> for
+            details about ranges, casting, and so on for the various numeric data types.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Most integer types from other systems have equivalents in Impala, perhaps under different names such as
+            <codeph>BIGINT</codeph> instead of <codeph>INT8</codeph>. For any that are unavailable, for example
+            <codeph>MEDIUMINT</codeph>, switch to the smallest Impala integer type that has sufficient range.
+            Remove any precision specifications. See <xref href="impala_datatypes.xml#datatypes"/> for details
+            about ranges, casting, and so on for the various numeric data types.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Remove any <codeph>UNSIGNED</codeph> constraints. All Impala numeric types are signed. See
+            <xref href="impala_datatypes.xml#datatypes"/> for details about ranges, casting, and so on for the
+            various numeric data types.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            For any types holding bitwise values, use an integer type with enough range to hold all the relevant
+            bits within a positive integer. See <xref href="impala_datatypes.xml#datatypes"/> for details about
+            ranges, casting, and so on for the various numeric data types.
+          </p>
+          <p>
+            For example, <codeph>TINYINT</codeph> has a maximum positive value of 127, not 256, so to manipulate
+            8-bit bitfields as positive numbers switch to the next largest type <codeph>SMALLINT</codeph>.
+          </p>
+<codeblock>[localhost:21000] &gt; select cast(127*2 as tinyint);
++--------------------------+
+| cast(127 * 2 as tinyint) |
++--------------------------+
+| -2                       |
++--------------------------+
+[localhost:21000] &gt; select cast(128 as tinyint);
++----------------------+
+| cast(128 as tinyint) |
++----------------------+
+| -128                 |
++----------------------+
+[localhost:21000] &gt; select cast(127*2 as smallint);
++---------------------------+
+| cast(127 * 2 as smallint) |
++---------------------------+
+| 254                       |
++---------------------------+</codeblock>
+          <p>
+            Impala does not support notation such as <codeph>b'0101'</codeph> for bit literals.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            For BLOB values, use <codeph>STRING</codeph> to represent <codeph>CLOB</codeph> or
+            <codeph>TEXT</codeph> types (character based large objects) up to 32 KB in size. Binary large objects
+            such as <codeph>BLOB</codeph>, <codeph>RAW</codeph> <codeph>BINARY</codeph>, and
+            <codeph>VARBINARY</codeph> do not currently have an equivalent in Impala.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            For Boolean-like types such as <codeph>BOOL</codeph>, use the Impala <codeph>BOOLEAN</codeph> type.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Because Impala currently does not support composite or nested types, any spatial data types in other
+            database systems do not have direct equivalents in Impala. You could represent spatial values in string
+            format and write UDFs to process them. See <xref href="impala_udf.xml#udfs"/> for details. Where
+            practical, separate spatial types into separate tables so that Impala can still work with the
+            non-spatial data.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Take out any <codeph>DEFAULT</codeph> clauses. Impala can use data files produced from many different
+            sources, such as Pig, Hive, or MapReduce jobs. The fast import mechanisms of <codeph>LOAD DATA</codeph>
+            and external tables mean that Impala is flexible about the format of data files, and Impala does not
+            necessarily validate or cleanse data before querying it. When copying data through Impala
+            <codeph>INSERT</codeph> statements, you can use conditional functions such as <codeph>CASE</codeph> or
+            <codeph>NVL</codeph> to substitute some other value for <codeph>NULL</codeph> fields; see
+            <xref href="impala_conditional_functions.xml#conditional_functions"/> for details.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Take out any constraints from your <codeph>CREATE TABLE</codeph> and <codeph>ALTER TABLE</codeph>
+            statements, for example <codeph>PRIMARY KEY</codeph>, <codeph>FOREIGN KEY</codeph>,
+            <codeph>UNIQUE</codeph>, <codeph>NOT NULL</codeph>, <codeph>UNSIGNED</codeph>, or
+            <codeph>CHECK</codeph> constraints. Impala can use data files produced from many different sources,
+            such as Pig, Hive, or MapReduce jobs. Therefore, Impala expects initial data validation to happen
+            earlier during the ETL or ELT cycle. After data is loaded into Impala tables, you can perform queries
+            to test for <codeph>NULL</codeph> values. When copying data through Impala <codeph>INSERT</codeph>
+            statements, you can use conditional functions such as <codeph>CASE</codeph> or <codeph>NVL</codeph> to
+            substitute some other value for <codeph>NULL</codeph> fields; see
+            <xref href="impala_conditional_functions.xml#conditional_functions"/> for details.
+          </p>
+          <p>
+            Do as much verification as practical before loading data into Impala. After data is loaded into Impala,
+            you can do further verification using SQL queries to check if values have expected ranges, if values
+            are <codeph>NULL</codeph> or not, and so on. If there is a problem with the data, you will need to
+            re-run earlier stages of the ETL process, or do an <codeph>INSERT ... SELECT</codeph> statement in
+            Impala to copy the faulty data to a new table and transform or filter out the bad values.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Take out any <codeph>CREATE INDEX</codeph>, <codeph>DROP INDEX</codeph>, and <codeph>ALTER
+            INDEX</codeph> statements, and equivalent <codeph>ALTER TABLE</codeph> statements. Remove any
+            <codeph>INDEX</codeph>, <codeph>KEY</codeph>, or <codeph>PRIMARY KEY</codeph> clauses from
+            <codeph>CREATE TABLE</codeph> and <codeph>ALTER TABLE</codeph> statements. Impala is optimized for bulk
+            read operations for data warehouse-style queries, and therefore does not support indexes for its
+            tables.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Calls to built-in functions with out-of-range or otherwise incorrect arguments, return
+            <codeph>NULL</codeph> in Impala as opposed to raising exceptions. (This rule applies even when the
+            <codeph>ABORT_ON_ERROR=true</codeph> query option is in effect.) Run small-scale queries using
+            representative data to doublecheck that calls to built-in functions are returning expected values
+            rather than <codeph>NULL</codeph>. For example, unsupported <codeph>CAST</codeph> operations do not
+            raise an error in Impala:
+          </p>
+<codeblock>select cast('foo' as int);
++--------------------+
+| cast('foo' as int) |
++--------------------+
+| NULL               |
++--------------------+</codeblock>
+        </li>
+
+        <li>
+          <p>
+            For any other type not supported in Impala, you could represent their values in string format and write
+            UDFs to process them. See <xref href="impala_udf.xml#udfs"/> for details.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            To detect the presence of unsupported or unconvertable data types in data files, do initial testing
+            with the <codeph>ABORT_ON_ERROR=true</codeph> query option in effect. This option causes queries to
+            fail immediately if they encounter disallowed type conversions. See
+            <xref href="impala_abort_on_error.xml#abort_on_error"/> for details. For example:
+          </p>
+<codeblock>set abort_on_error=true;
+select count(*) from (select * from t1);
+-- The above query will fail if the data files for T1 contain any
+-- values that can't be converted to the expected Impala data types.
+-- For example, if T1.C1 is defined as INT but the column contains
+-- floating-point values like 1.1, the query will return an error.</codeblock>
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="porting_statements">
+
+    <title>SQL Statements to Remove or Adapt</title>
+
+    <conbody>
+
+      <p>
+        Some SQL statements or clauses that you might be familiar with are not currently supported in Impala:
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            Impala has no <codeph>DELETE</codeph> statement. Impala is intended for data warehouse-style operations
+            where you do bulk moves and transforms of large quantities of data. Instead of using
+            <codeph>DELETE</codeph>, use <codeph>INSERT OVERWRITE</codeph> to entirely replace the contents of a
+            table or partition, or use <codeph>INSERT ... SELECT</codeph> to copy a subset of data (everything but
+            the rows you intended to delete) from one table to another. See <xref href="impala_dml.xml#dml"/> for
+            an overview of Impala DML statements.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Impala has no <codeph>UPDATE</codeph> statement. Impala is intended for data warehouse-style operations
+            where you do bulk moves and transforms of large quantities of data. Instead of using
+            <codeph>UPDATE</codeph>, do all necessary transformations early in the ETL process, such as in the job
+            that generates the original data, or when copying from one table to another to convert to a particular
+            file format or partitioning scheme. See <xref href="impala_dml.xml#dml"/> for an overview of Impala DML
+            statements.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Impala has no transactional statements, such as <codeph>COMMIT</codeph> or <codeph>ROLLBACK</codeph>.
+            Impala effectively works like the <codeph>AUTOCOMMIT</codeph> mode in some database systems, where
+            changes take effect as soon as they are made.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            If your database, table, column, or other names conflict with Impala reserved words, use different
+            names or quote the names with backticks. See <xref href="impala_reserved_words.xml#reserved_words"/>
+            for the current list of Impala reserved words.
+          </p>
+          <p>
+            Conversely, if you use a keyword that Impala does not recognize, it might be interpreted as a table or
+            column alias. For example, in <codeph>SELECT * FROM t1 NATURAL JOIN t2</codeph>, Impala does not
+            recognize the <codeph>NATURAL</codeph> keyword and interprets it as an alias for the table
+            <codeph>t1</codeph>. If you experience any unexpected behavior with queries, check the list of reserved
+            words to make sure all keywords in join and <codeph>WHERE</codeph> clauses are recognized.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Impala supports subqueries only in the <codeph>FROM</codeph> clause of a query, not within the
+            <codeph>WHERE</codeph> clauses. Therefore, you cannot use clauses such as <codeph>WHERE
+            <varname>column</varname> IN (<varname>subquery</varname>)</codeph>. Also, Impala does not allow
+            <codeph>EXISTS</codeph> or <codeph>NOT EXISTS</codeph> clauses (although <codeph>EXISTS</codeph> is a
+            reserved keyword).
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Impala supports <codeph>UNION</codeph> and <codeph>UNION ALL</codeph> set operators, but not
+            <codeph>INTERSECT</codeph>. <ph conref="../shared/impala_common.xml#common/union_all_vs_union"/>
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Within queries, Impala requires query aliases for any subqueries:
+          </p>
+<codeblock>-- Without the alias 'contents_of_t1' at the end, query gives syntax error.
+select count(*) from (select * from t1) contents_of_t1;</codeblock>
+        </li>
+
+        <li>
+          <p>
+            When an alias is declared for an expression in a query, that alias cannot be referenced again within
+            the same query block:
+          </p>
+<codeblock>-- Can't reference AVERAGE twice in the SELECT list where it's defined.
+select avg(x) as average, average+1 from t1 group by x;
+ERROR: AnalysisException: couldn't resolve column reference: 'average'
+
+-- Although it can be referenced again later in the same query.
+select avg(x) as average from t1 group by x having average &gt; 3;</codeblock>
+          <p>
+            For Impala, either repeat the expression again, or abstract the expression into a <codeph>WITH</codeph>
+            clause, creating named columns that can be referenced multiple times anywhere in the base query:
+          </p>
+<codeblock>-- The following 2 query forms are equivalent.
+select avg(x) as average, avg(x)+1 from t1 group by x;
+with avg_t as (select avg(x) average from t1 group by x) select average, average+1 from avg_t;</codeblock>
+<!-- An alternative bunch of queries to use in the example above.
+[localhost:21000] > select x*x as x_squared from t1;
+
+[localhost:21000] > select x*x as x_squared from t1 where x_squared < 100;
+ERROR: AnalysisException: couldn't resolve column reference: 'x_squared'
+[localhost:21000] > select x*x as x_squared, x_squared * pi() as pi_x_squared from t1;
+ERROR: AnalysisException: couldn't resolve column reference: 'x_squared'
+[localhost:21000] > select x*x as x_squared from t1 group by x_squared;
+
+[localhost:21000] > select x*x as x_squared from t1 group by x_squared having x_squared < 100;
+-->
+        </li>
+
+        <li>
+          <p>
+            Impala does not support certain rarely used join types that are less appropriate for high-volume tables
+            used for data warehousing. In some cases, Impala supports join types but requires explicit syntax to
+            ensure you do not do inefficient joins of huge tables by accident. For example, Impala does not support
+            natural joins or anti-joins, and requires the <codeph>CROSS JOIN</codeph> operator for Cartesian
+            products. See <xref href="impala_joins.xml#joins"/> for details on the syntax for Impala join clauses.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Impala has a limited choice of partitioning types. Partitions are defined based on each distinct
+            combination of values for one or more partition key columns. Impala does not redistribute or check data
+            to create evenly distributed partitions; you must choose partition key columns based on your knowledge
+            of the data volume and distribution. Adapt any tables that use range, list, hash, or key partitioning
+            to use the Impala partition syntax for <codeph>CREATE TABLE</codeph> and <codeph>ALTER TABLE</codeph>
+            statements. Impala partitioning is similar to range partitioning where every range has exactly one
+            value, or key partitioning where the hash function produces a separate bucket for every combination of
+            key values. See <xref href="impala_partitioning.xml#partitioning"/> for usage details, and
+            <xref href="impala_create_table.xml#create_table"/> and
+            <xref href="impala_alter_table.xml#alter_table"/> for syntax.
+          </p>
+          <note>
+            Because the number of separate partitions is potentially higher than in other database systems, keep a
+            close eye on the number of partitions and the volume of data in each one; scale back the number of
+            partition key columns if you end up with too many partitions with a small volume of data in each one.
+            Remember, to distribute work for a query across a cluster, you need at least one HDFS block per node.
+            HDFS blocks are typically multiple megabytes, <ph rev="parquet_block_size">especially</ph> for Parquet
+            files. Therefore, if each partition holds only a few megabytes of data, you are unlikely to see much
+            parallelism in the query because such a small amount of data is typically processed by a single node.
+          </note>
+        </li>
+
+        <li>
+          <p>
+            For <q>top-N</q> queries, Impala uses the <codeph>LIMIT</codeph> clause rather than comparing against a
+            pseudocolumn named <codeph>ROWNUM</codeph> or <codeph>ROW_NUM</codeph>. See
+            <xref href="impala_limit.xml#limit"/> for details.
+          </p>
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="porting_antipatterns">
+
+    <title>SQL Constructs to Doublecheck</title>
+
+    <conbody>
+
+      <p>
+        Some SQL constructs that are supported have behavior or defaults more oriented towards convenience than
+        optimal performance. Also, sometimes machine-generated SQL, perhaps issued through JDBC or ODBC
+        applications, might have inefficiencies or exceed internal Impala limits. As you port SQL code, be alert
+        and change these things where appropriate:
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            A <codeph>CREATE TABLE</codeph> statement with no <codeph>STORED AS</codeph> clause creates data files
+            in plain text format, which is convenient for data interchange but not a good choice for high-volume
+            data with high-performance queries. See <xref href="impala_file_formats.xml#file_formats"/> for why and
+            how to use specific file formats for compact data and high-performance queries. Especially see
+            <xref href="impala_parquet.xml#parquet"/>, for details about the file format most heavily optimized for
+            large-scale data warehouse queries.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            A <codeph>CREATE TABLE</codeph> statement with no <codeph>PARTITIONED BY</codeph> clause stores all the
+            data files in the same physical location, which can lead to scalability problems when the data volume
+            becomes large.
+          </p>
+          <p>
+            On the other hand, adapting tables that were already partitioned in a different database system could
+            produce an Impala table with a high number of partitions and not enough data in each one, leading to
+            underutilization of Impala's parallel query features.
+          </p>
+          <p>
+            See <xref href="impala_partitioning.xml#partitioning"/> for details about setting up partitioning and
+            tuning the performance of queries on partitioned tables.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            The <codeph>INSERT ... VALUES</codeph> syntax is suitable for setting up toy tables with a few rows for
+            functional testing, but because each such statement creates a separate tiny file in HDFS, it is not a
+            scalable technique for loading megabytes or gigabytes (let alone petabytes) of data. Consider revising
+            your data load process to produce raw data files outside of Impala, then setting up Impala external
+            tables or using the <codeph>LOAD DATA</codeph> statement to use those data files instantly in Impala
+            tables, with no conversion or indexing stage. See <xref href="impala_tables.xml#external_tables"/> and
+            <xref href="impala_load_data.xml#load_data"/> for details about the Impala techniques for working with
+            data files produced outside of Impala; see <xref href="impala_tutorial.xml#tutorial_etl"/> for examples
+            of ETL workflow for Impala.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            If your ETL process is not optimized for Hadoop, you might end up with highly fragmented small data
+            files, or a single giant data file that cannot take advantage of distributed parallel queries or
+            partitioning. In this case, use an <codeph>INSERT ... SELECT</codeph> statement to copy the data into a
+            new table and reorganize into a more efficient layout in the same operation. See
+            <xref href="impala_insert.xml#insert"/> for details about the <codeph>INSERT</codeph> statement.
+          </p>
+          <p>
+            You can do <codeph>INSERT ... SELECT</codeph> into a table with a more efficient file format (see
+            <xref href="impala_file_formats.xml#file_formats"/>) or from an unpartitioned table into a partitioned
+            one (see <xref href="impala_partitioning.xml#partitioning"/>).
+          </p>
+        </li>
+
+        <li>
+          <p>
+            The number of expressions allowed in an Impala query might be smaller than for some other database
+            systems, causing failures for very complicated queries (typically produced by automated SQL
+            generators). Where practical, keep the number of expressions in the <codeph>WHERE</codeph> clauses to
+            approximately 2000 or fewer. As a workaround, set the query option
+            <codeph>DISABLE_CODEGEN=true</codeph> if queries fail for this reason. See
+            <xref href="impala_disable_codegen.xml#disable_codegen"/> for details.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            If practical, rewrite <codeph>UNION</codeph> queries to use the <codeph>UNION ALL</codeph> operator
+            instead. <ph conref="../shared/impala_common.xml#common/union_all_vs_union"/>
+          </p>
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="porting_next">
+
+    <title>Next Porting Steps after Verifying Syntax and Semantics</title>
+
+    <conbody>
+
+      <p>
+        Throughout this section, some of the decisions you make during the porting process also have a substantial
+        impact on performance. After your SQL code is ported and working correctly, doublecheck the
+        performance-related aspects of your schema design, physical layout, and queries to make sure that the
+        ported application is taking full advantage of Impala's parallelism, performance-related SQL features, and
+        integration with Hadoop components.
+      </p>
+
+      <ul>
+        <li>
+          Have you run the <codeph>COMPUTE STATS</codeph> statement on each table involved in join queries? Have
+          you also run <codeph>COMPUTE STATS</codeph> for each table used as the source table in an <codeph>INSERT
+          ... SELECT</codeph> or <codeph>CREATE TABLE AS SELECT</codeph> statement?
+        </li>
+
+        <li>
+          Are you using the most efficient file format for your data volumes, table structure, and query
+          characteristics?
+        </li>
+
+        <li>
+          Are you using partitioning effectively? That is, have you partitioned on columns that are often used for
+          filtering in <codeph>WHERE</codeph> clauses? Have you partitioned at the right granularity so that there
+          is enough data in each partition to parallelize the work for each query?
+        </li>
+
+        <li>
+          Does your ETL process produce a relatively small number of multi-megabyte data files (good) rather than a
+          huge number of small files (bad)?
+        </li>
+      </ul>
+
+      <p>
+        See <xref href="impala_performance.xml#performance"/> for details about the whole performance tuning
+        process.
+      </p>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_ports.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_ports.xml b/docs/topics/impala_ports.xml
new file mode 100644
index 0000000..80f217f
--- /dev/null
+++ b/docs/topics/impala_ports.xml
@@ -0,0 +1,440 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="ports">
+
+  <title>Ports Used by Impala</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Ports"/>
+      <data name="Category" value="Network"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody id="conbody_ports">
+
+    <p>
+      <indexterm audience="Cloudera">ports</indexterm>
+      Impala uses the TCP ports listed in the following table. Before deploying Impala, ensure these ports are open
+      on each system.
+    </p>
+
+    <table>
+      <tgroup cols="5">
+        <colspec colname="1" colwidth="20*"/>
+        <colspec colname="2" colwidth="30*"/>
+        <colspec colname="3" colwidth="10*"/>
+        <colspec colname="4" colwidth="20*"/>
+        <colspec colname="5" colwidth="30*"/>
+        <thead>
+          <row>
+            <entry>
+              Component
+            </entry>
+            <entry>
+              Service
+            </entry>
+            <entry>
+              Port
+            </entry>
+            <entry>
+              Access Requirement
+            </entry>
+            <entry>
+              Comment
+            </entry>
+          </row>
+        </thead>
+        <tbody>
+          <row>
+            <entry>
+              <p>
+                Impala Daemon
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Impala Daemon Frontend Port
+              </p>
+            </entry>
+            <entry>
+              <p>
+                21000
+              </p>
+            </entry>
+            <entry>
+              <p>
+                External
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Used to transmit commands and receive results by <codeph>impala-shell</codeph> and
+                version 1.2 of the Cloudera ODBC driver.
+              </p>
+            </entry>
+          </row>
+          <row>
+            <entry>
+              <p>
+                Impala Daemon
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Impala Daemon Frontend Port
+              </p>
+            </entry>
+            <entry>
+              <p>
+                21050
+              </p>
+            </entry>
+            <entry>
+              <p>
+                External
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Used to transmit commands and receive results by applications, such as Business Intelligence tools,
+                using JDBC, the Beeswax query editor in Hue, and version 2.0 or higher of the Cloudera ODBC driver.
+              </p>
+            </entry>
+          </row>
+          <row>
+            <entry>
+              <p>
+                Impala Daemon
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Impala Daemon Backend Port
+              </p>
+            </entry>
+            <entry>
+              <p>
+                22000
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Internal
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Internal use only. Impala daemons use this port to communicate with each other.
+              </p>
+            </entry>
+          </row>
+          <row>
+            <entry>
+              <p>
+                Impala Daemon
+              </p>
+            </entry>
+            <entry>
+              <p>
+                StateStoreSubscriber Service Port
+              </p>
+            </entry>
+            <entry>
+              <p>
+                23000
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Internal
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Internal use only. Impala daemons listen on this port for updates from the statestore daemon.
+              </p>
+            </entry>
+          </row>
+          <row rev="2.1.0">
+            <entry>
+              <p>
+                Catalog Daemon
+              </p>
+            </entry>
+            <entry>
+              <p>
+                StateStoreSubscriber Service Port
+              </p>
+            </entry>
+            <entry>
+              <p>
+                23020
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Internal
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Internal use only. The catalog daemon listens on this port for updates from the statestore daemon.
+              </p>
+            </entry>
+          </row>
+          <row>
+            <entry>
+              <p>
+                Impala Daemon
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Impala Daemon HTTP Server Port
+              </p>
+            </entry>
+            <entry>
+              <p>
+                25000
+              </p>
+            </entry>
+            <entry>
+              <p>
+                External
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Impala web interface for administrators to monitor and troubleshoot.
+              </p>
+            </entry>
+          </row>
+          <row>
+            <entry>
+              <p>
+                Impala StateStore Daemon
+              </p>
+            </entry>
+            <entry>
+              <p>
+                StateStore HTTP Server Port
+              </p>
+            </entry>
+            <entry>
+              <p>
+                25010
+              </p>
+            </entry>
+            <entry>
+              <p>
+                External
+              </p>
+            </entry>
+            <entry>
+              <p>
+                StateStore web interface for administrators to monitor and troubleshoot.
+              </p>
+            </entry>
+          </row>
+          <row rev="1.2">
+            <entry>
+              <p>
+                Impala Catalog Daemon
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Catalog HTTP Server Port
+              </p>
+            </entry>
+            <entry>
+              <p>
+                25020
+              </p>
+            </entry>
+            <entry>
+              <p>
+                External
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Catalog service web interface for administrators to monitor and troubleshoot. New in Impala 1.2 and
+                higher.
+              </p>
+            </entry>
+          </row>
+          <row>
+            <entry>
+              <p>
+                Impala StateStore Daemon
+              </p>
+            </entry>
+            <entry>
+              <p>
+                StateStore Service Port
+              </p>
+            </entry>
+            <entry>
+              <p>
+                24000
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Internal
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Internal use only. The statestore daemon listens on this port for registration/unregistration
+                requests.
+              </p>
+            </entry>
+          </row>
+          <row rev="1.2">
+            <entry>
+              <p>
+                Impala Catalog Daemon
+              </p>
+            </entry>
+            <entry>
+              <p>
+                StateStore Service Port
+              </p>
+            </entry>
+            <entry>
+              <p>
+                26000
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Internal
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Internal use only. The catalog service uses this port to communicate with the Impala daemons. New
+                in Impala 1.2 and higher.
+              </p>
+            </entry>
+          </row>
+          <row rev="1.3.0">
+            <entry>
+              <p>
+                Impala Daemon
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Llama Callback Port
+              </p>
+            </entry>
+            <entry>
+              <p>
+                28000
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Internal
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Internal use only. Impala daemons use to communicate with Llama. New in <ph rev="upstream">CDH 5.0.0</ph> and higher.
+              </p>
+            </entry>
+          </row>
+          <row rev="1.3.0">
+            <entry>
+              <p>
+                Impala Llama ApplicationMaster
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Llama Thrift Admin Port
+              </p>
+            </entry>
+            <entry>
+              <p>
+                15002
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Internal
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Internal use only. New in <ph rev="upstream">CDH 5.0.0</ph> and higher.
+              </p>
+            </entry>
+          </row>
+          <row rev="1.3.0">
+            <entry>
+              <p>
+                Impala Llama ApplicationMaster
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Llama Thrift Port
+              </p>
+            </entry>
+            <entry>
+              <p>
+                15000
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Internal
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Internal use only. New in <ph rev="upstream">CDH 5.0.0</ph> and higher.
+              </p>
+            </entry>
+          </row>
+          <row rev="1.3.0">
+            <entry>
+              <p>
+                Impala Llama ApplicationMaster
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Llama HTTP Port
+              </p>
+            </entry>
+            <entry>
+              <p>
+                15001
+              </p>
+            </entry>
+            <entry>
+              <p>
+                External
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Llama service web interface for administrators to monitor and troubleshoot. New in CDH 5.0.0 and
+                higher.
+              </p>
+            </entry>
+          </row>
+        </tbody>
+      </tgroup>
+    </table>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_prefetch_mode.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_prefetch_mode.xml b/docs/topics/impala_prefetch_mode.xml
new file mode 100644
index 0000000..fc85c11
--- /dev/null
+++ b/docs/topics/impala_prefetch_mode.xml
@@ -0,0 +1,49 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="prefetch_mode" rev="2.6.0 IMPALA-3286">
+
+  <title>PREFETCH_MODE Query Option (<keyword keyref="impala26"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>PREFETCH_MODE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.6.0 IMPALA-3286">
+      <indexterm audience="Cloudera">PREFETCH_MODE query option</indexterm>
+      Determines whether the prefetching optimization is applied during
+      join query processing.
+    </p>
+
+    <p>
+      <b>Type:</b> numeric (0, 1)
+      or corresponding mnemonic strings (<codeph>NONE</codeph>, <codeph>HT_BUCKET</codeph>).
+    </p>
+
+    <p>
+      <b>Default:</b> 1 (equivalent to <codeph>HT_BUCKET</codeph>)
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/added_in_260"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+    <p>
+      The default mode is 1, which means that hash table buckets are
+      prefetched during join query processing.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_joins.xml#joins"/>,
+      <xref href="impala_perf_joins.xml#perf_joins"/>.
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_prereqs.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_prereqs.xml b/docs/topics/impala_prereqs.xml
new file mode 100644
index 0000000..8572738
--- /dev/null
+++ b/docs/topics/impala_prereqs.xml
@@ -0,0 +1,357 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="prereqs">
+
+  <title>Impala Requirements</title>
+  <titlealts audience="PDF"><navtitle>Requirements</navtitle></titlealts>
+
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Requirements"/>
+      <data name="Category" value="Planning"/>
+      <data name="Category" value="Installing"/>
+      <data name="Category" value="Upgrading"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+      <!-- Another instance of a topic pulled into the map twice, resulting in a second HTML page with a *1.html filename. -->
+      <data name="Category" value="Duplicate Topics"/>
+      <!-- Using a separate category, 'Multimap', to flag those pages that are duplicate because of multiple DITA map references. -->
+      <data name="Category" value="Multimap"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">prerequisites</indexterm>
+      <indexterm audience="Cloudera">requirements</indexterm>
+      To perform as expected, Impala depends on the availability of the software, hardware, and configurations
+      described in the following sections.
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="product_compatibility_matrix">
+
+    <title>Product Compatibility Matrix</title>
+
+    <conbody>
+
+      <p> The ultimate source of truth about compatibility between various
+        versions of CDH, Cloudera Manager, and various CDH components is the <ph
+          audience="integrated"><xref
+            href="rn_consolidated_pcm.xml"
+            >Product Compatibility Matrix for CDH and Cloudera
+          Manager</xref></ph><ph audience="standalone">online <xref
+            href="http://www.cloudera.com/documentation/enterprise/latest/topics/rn_consolidated_pcm.html"
+            format="html" scope="external">Product Compatibility
+          Matrix</xref></ph>. </p>
+
+      <p>
+        For Impala, see the
+        <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/pcm_impala.html" scope="external" format="html">Impala
+        compatibility matrix page</xref>.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="prereqs_os">
+
+    <title>Supported Operating Systems</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">software requirements</indexterm>
+        <indexterm audience="Cloudera">Red Hat Enterprise Linux</indexterm>
+        <indexterm audience="Cloudera">RHEL</indexterm>
+        <indexterm audience="Cloudera">CentOS</indexterm>
+        <indexterm audience="Cloudera">SLES</indexterm>
+        <indexterm audience="Cloudera">Ubuntu</indexterm>
+        <indexterm audience="Cloudera">SUSE</indexterm>
+        <indexterm audience="Cloudera">Debian</indexterm> The relevant supported operating systems
+        and versions for Impala are the same as for the corresponding CDH 5 platforms. For
+        details, see the <cite>Supported Operating Systems</cite> page for
+        <ph audience="integrated"><xref href="rn_consolidated_pcm.xml#cdh_cm_supported_os">CDH
+            5</xref></ph><ph audience="standalone"><xref
+            href="http://www.cloudera.com/documentation/enterprise/latest/topics/rn_consolidated_pcm.html#cdh_cm_supported_os"
+            scope="external" format="html">CDH 5</xref></ph>. </p>
+    </conbody>
+  </concept>
+
+  <concept id="prereqs_hive">
+
+    <title>Hive Metastore and Related Configuration</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Metastore"/>
+      <data name="Category" value="Hive"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">Hive</indexterm>
+        <indexterm audience="Cloudera">MySQL</indexterm>
+        <indexterm audience="Cloudera">PostgreSQL</indexterm>
+        Impala can interoperate with data stored in Hive, and uses the same infrastructure as Hive for tracking
+        metadata about schema objects such as tables and columns. The following components are prerequisites for
+        Impala:
+      </p>
+
+      <ul>
+        <li>
+          MySQL or PostgreSQL, to act as a metastore database for both Impala and Hive.
+          <note>
+            <p>
+              Installing and configuring a Hive metastore is an Impala requirement. Impala does not work without
+              the metastore database. For the process of installing and configuring the metastore, see
+              <xref href="impala_install.xml#install"/>.
+            </p>
+            <p>
+              Always configure a <b>Hive metastore service</b> rather than connecting directly to the metastore
+              database. The Hive metastore service is required to interoperate between possibly different levels of
+              metastore APIs used by CDH and Impala, and avoids known issues with connecting directly to the
+              metastore database. The Hive metastore service is set up for you by default if you install through
+              Cloudera Manager 4.5 or higher.
+            </p>
+            <p>
+              A summary of the metastore installation process is as follows:
+            </p>
+            <ul>
+              <li>
+                Install a MySQL or PostgreSQL database. Start the database if it is not started after installation.
+              </li>
+
+              <li>
+                Download the
+                <xref href="http://www.mysql.com/products/connector/" scope="external" format="html">MySQL
+                connector</xref> or the
+                <xref href="http://jdbc.postgresql.org/download.html" scope="external" format="html">PostgreSQL
+                connector</xref> and place it in the <codeph>/usr/share/java/</codeph> directory.
+              </li>
+
+              <li>
+                Use the appropriate command line tool for your database to create the metastore database.
+              </li>
+
+              <li>
+                Use the appropriate command line tool for your database to grant privileges for the metastore
+                database to the <codeph>hive</codeph> user.
+              </li>
+
+              <li>
+                Modify <codeph>hive-site.xml</codeph> to include information matching your particular database: its
+                URL, username, and password. You will copy the <codeph>hive-site.xml</codeph> file to the Impala
+                Configuration Directory later in the Impala installation process.
+              </li>
+            </ul>
+          </note>
+        </li>
+
+        <li>
+          <b>Optional:</b> Hive. Although only the Hive metastore database is required for Impala to function, you
+          might install Hive on some client machines to create and load data into tables that use certain file
+          formats. See <xref href="impala_file_formats.xml#file_formats"/> for details. Hive does not need to be
+          installed on the same DataNodes as Impala; it just needs access to the same metastore database.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="prereqs_java">
+
+    <title>Java Dependencies</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Java"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">Java</indexterm>
+        <indexterm audience="Cloudera">impala-dependencies.jar</indexterm>
+        Although Impala is primarily written in C++, it does use Java to communicate with various Hadoop
+        components:
+      </p>
+
+      <ul>
+        <li>
+          The officially supported JVM for Impala is the Oracle JVM. Other JVMs might cause issues, typically
+          resulting in a failure at <cmdname>impalad</cmdname> startup. In particular, the JamVM used by default on
+          certain levels of Ubuntu systems can cause <cmdname>impalad</cmdname> to fail to start.
+          <!-- To do:
+            Could say something here about JDK 6 vs. JDK 7 in CDH 5. Since we didn't specify the JDK version before,
+            don't know the impact from the user perspective so not calling it out at the moment.
+          -->
+        </li>
+
+        <li>
+          Internally, the <cmdname>impalad</cmdname> daemon relies on the <codeph>JAVA_HOME</codeph> environment
+          variable to locate the system Java libraries. Make sure the <cmdname>impalad</cmdname> service is not run
+          from an environment with an incorrect setting for this variable.
+        </li>
+
+        <li>
+          All Java dependencies are packaged in the <codeph>impala-dependencies.jar</codeph> file, which is located
+          at <codeph>/usr/lib/impala/lib/</codeph>. These map to everything that is built under
+          <codeph>fe/target/dependency</codeph>.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="prereqs_network">
+
+    <title>Networking Configuration Requirements</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Network"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">network configuration</indexterm>
+        As part of ensuring best performance, Impala attempts to complete tasks on local data, as opposed to using
+        network connections to work with remote data. To support this goal, Impala matches
+        the�<b>hostname</b>�provided to each Impala daemon with the�<b>IP address</b>�of each DataNode by
+        resolving the hostname flag to an IP address. For Impala to work with local data, use a single IP interface
+        for the DataNode and the Impala daemon on each machine. Ensure that the Impala daemon's hostname flag
+        resolves to the IP address of the DataNode. For single-homed machines, this is usually automatic, but for
+        multi-homed machines, ensure that the Impala daemon's hostname resolves to the correct interface. Impala
+        tries to detect the correct hostname at start-up, and prints the derived hostname at the start of the log
+        in a message of the form:
+      </p>
+
+<codeblock>Using hostname: impala-daemon-1.example.com</codeblock>
+
+      <p>
+        In the majority of cases, this automatic detection works correctly. If you need to explicitly set the
+        hostname, do so by setting the�<codeph>--hostname</codeph>�flag.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="prereqs_hardware">
+
+    <title>Hardware Requirements</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">hardware requirements</indexterm>
+        <indexterm audience="Cloudera">capacity</indexterm>
+        <indexterm audience="Cloudera">RAM</indexterm>
+        <indexterm audience="Cloudera">memory</indexterm>
+        <indexterm audience="Cloudera">CPU</indexterm>
+        <indexterm audience="Cloudera">processor</indexterm>
+        <indexterm audience="Cloudera">Intel</indexterm>
+        <indexterm audience="Cloudera">AMD</indexterm>
+        During join operations, portions of data from each joined table are loaded into memory. Data sets can be
+        very large, so ensure your hardware has sufficient memory to accommodate the joins you anticipate
+        completing.
+      </p>
+
+      <p>
+        While requirements vary according to data set size, the following is generally recommended:
+      </p>
+
+      <ul>
+        <li rev="2.0.0">
+          CPU - Impala version 2.2 and higher uses the SSSE3 instruction set, which is included in newer processors.
+          <note>
+            This required level of processor is the same as in Impala version 1.x. The Impala 2.0 and 2.1 releases
+            had a stricter requirement for the SSE4.1 instruction set, which has now been relaxed.
+          </note>
+<!--
+          For best performance use:
+          <ul>
+            <li>
+              Intel - Nehalem (released 2008) or later processors.
+            </li>
+
+            <li>
+              AMD - Bulldozer (released 2011) or later processors.
+            </li>
+          </ul>
+-->
+        </li>
+
+        <li rev="1.2">
+          Memory - 128 GB or more recommended, ideally 256 GB or more. If the intermediate results during query
+          processing on a particular node exceed the amount of memory available to Impala on that node, the query
+          writes temporary work data to disk, which can lead to long query times. Note that because the work is
+          parallelized, and intermediate results for aggregate queries are typically smaller than the original
+          data, Impala can query and join tables that are much larger than the memory available on an individual
+          node.
+        </li>
+
+        <li>
+          Storage - DataNodes with 12 or more disks each. I/O speeds are often the limiting factor for disk
+          performance with Impala. Ensure that you have sufficient disk space to store the data Impala will be
+          querying.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="prereqs_account">
+
+    <title>User Account Requirements</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Users"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">impala user</indexterm>
+        <indexterm audience="Cloudera">impala group</indexterm>
+        <indexterm audience="Cloudera">root user</indexterm>
+        Impala creates and uses a user and group named <codeph>impala</codeph>. Do not delete this account or group
+        and do not modify the account's or group's permissions and rights. Ensure no existing systems obstruct the
+        functioning of these accounts and groups. For example, if you have scripts that delete user accounts not in
+        a white-list, add these accounts to the list of permitted accounts.
+      </p>
+
+<!-- Taking out because no longer applicable in CDH 5.5 and up. -->
+      <p id="impala_hdfs_group" rev="1.2" audience="Cloudera">
+        For the resource management feature to work (in combination with CDH 5 and the YARN and Llama components),
+        the <codeph>impala</codeph> user must be a member of the <codeph>hdfs</codeph> group. This setup is
+        performed automatically during a new install, but not when upgrading from earlier Impala releases to Impala
+        1.2. If you are upgrading a node to CDH 5 that already had Impala 1.1 or 1.0 installed, manually add the
+        <codeph>impala</codeph> user to the <codeph>hdfs</codeph> group.
+      </p>
+
+      <p>
+        For correct file deletion during <codeph>DROP TABLE</codeph> operations, Impala must be able to move files
+        to the HDFS trashcan. You might need to create an HDFS directory <filepath>/user/impala</filepath>,
+        writeable by the <codeph>impala</codeph> user, so that the trashcan can be created. Otherwise, data files
+        might remain behind after a <codeph>DROP TABLE</codeph> statement.
+      </p>
+
+      <p>
+        Impala should not run as root. Best Impala performance is achieved using direct reads, but root is not
+        permitted to use direct reads. Therefore, running Impala as root negatively affects performance.
+      </p>
+
+      <p>
+        By default, any user can connect to Impala and access all the associated databases and tables. You can
+        enable authorization and authentication based on the Linux OS user who connects to the Impala server, and
+        the associated groups for that user. <xref href="impala_security.xml#security"/> for details. These
+        security features do not change the underlying file permission requirements; the <codeph>impala</codeph>
+        user still needs to be able to access the data files.
+      </p>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_processes.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_processes.xml b/docs/topics/impala_processes.xml
new file mode 100644
index 0000000..05f2274
--- /dev/null
+++ b/docs/topics/impala_processes.xml
@@ -0,0 +1,134 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="processes">
+
+  <title>Starting Impala</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Starting and Stopping"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Operators"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="1.2">
+      <indexterm audience="Cloudera">state store</indexterm>
+      <indexterm audience="Cloudera">starting services</indexterm>
+      <indexterm audience="Cloudera">services</indexterm>
+      To activate Impala if it is installed but not yet started:
+    </p>
+
+    <ol>
+      <li>
+        Set any necessary configuration options for the Impala services. See
+        <xref href="impala_config_options.xml#config_options"/> for details.
+      </li>
+
+      <li>
+        Start one instance of the Impala statestore. The statestore helps Impala to distribute work efficiently,
+        and to continue running in the event of availability problems for other Impala nodes. If the statestore
+        becomes unavailable, Impala continues to function.
+      </li>
+
+      <li>
+        Start one instance of the Impala catalog service.
+      </li>
+
+      <li>
+        Start the main Impala service on one or more DataNodes, ideally on all DataNodes to maximize local
+        processing and avoid network traffic due to remote reads.
+      </li>
+    </ol>
+
+    <p>
+      Once Impala is running, you can conduct interactive experiments using the instructions in
+      <xref href="impala_tutorial.xml#tutorial"/> and try <xref href="impala_impala_shell.xml#impala_shell"/>.
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="starting_via_cm">
+
+    <title>Starting Impala through Cloudera Manager</title>
+
+    <conbody>
+
+      <p>
+        If you installed Impala with Cloudera Manager, use Cloudera Manager to start and stop services. The
+        Cloudera Manager GUI is a convenient way to check that all services are running, to set configuration
+        options using form fields in a browser, and to spot potential issues such as low disk space before they
+        become serious. Cloudera Manager automatically starts all the Impala-related services as a group, in the
+        correct order. See
+        <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_start_stop_service.html" scope="external" format="html">the
+        Cloudera Manager Documentation</xref> for details.
+      </p>
+
+      <note>
+        <p conref="../shared/impala_common.xml#common/udf_persistence_restriction"/>
+      </note>
+    </conbody>
+  </concept>
+
+  <concept id="starting_via_cmdline">
+
+    <title>Starting Impala from the Command Line</title>
+
+    <conbody>
+
+      <p>
+        To start the Impala state store and Impala from the command line or a script, you can either use the
+        <cmdname>service</cmdname> command or you can start the daemons directly through the
+        <cmdname>impalad</cmdname>, <codeph>statestored</codeph>, and <cmdname>catalogd</cmdname> executables.
+      </p>
+
+      <p>
+        Start the Impala statestore and then start <codeph>impalad</codeph> instances. You can modify the values
+        the service initialization scripts use when starting the statestore and Impala by editing
+        <codeph>/etc/default/impala</codeph>.
+      </p>
+
+      <p>
+        Start the statestore service using a command similar to the following:
+      </p>
+
+      <p>
+<codeblock>$ sudo service impala-state-store start</codeblock>
+      </p>
+
+      <p rev="1.2">
+        Start the catalog service using a command similar to the following:
+      </p>
+
+<codeblock rev="1.2">$ sudo service impala-catalog start</codeblock>
+
+      <p>
+        Start the Impala service on each DataNode using a command similar to the following:
+      </p>
+
+      <p>
+<codeblock>$ sudo service impala-server start</codeblock>
+      </p>
+
+      <note>
+        <p conref="../shared/impala_common.xml#common/udf_persistence_restriction"/>
+      </note>
+
+      <p>
+        If any of the services fail to start, review:
+        <ul>
+          <li>
+            <xref href="impala_logging.xml#logs_debug"/>
+          </li>
+
+          <li>
+            <xref href="impala_troubleshooting.xml#troubleshooting"/>
+          </li>
+        </ul>
+      </p>
+    </conbody>
+  </concept>
+</concept>


[40/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_complex_types.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_complex_types.xml b/docs/topics/impala_complex_types.xml
new file mode 100644
index 0000000..105ecdd
--- /dev/null
+++ b/docs/topics/impala_complex_types.xml
@@ -0,0 +1,2737 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="2.3.0" id="complex_types">
+
+  <title id="nested_types">Complex Types (<keyword keyref="impala23"/> or higher only)</title>
+
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.3.0">
+      <indexterm audience="Cloudera">complex types</indexterm>
+
+      <indexterm audience="Cloudera">nested types</indexterm>
+      <term>Complex types</term> (also referred to as <term>nested types</term>) let you represent multiple data values within a single
+      row/column position. They differ from the familiar column types such as <codeph>BIGINT</codeph> and <codeph>STRING</codeph>, known as
+      <term>scalar types</term> or <term>primitive types</term>, which represent a single data value within a given row/column position.
+      Impala supports the complex types <codeph>ARRAY</codeph>, <codeph>MAP</codeph>, and <codeph>STRUCT</codeph> in CDH 5.5 / Impala 2.3
+      and higher. The Hive <codeph>UNION</codeph> type is not currently supported.
+    </p>
+
+    <p outputclass="toc inpage"/>
+
+    <p>
+      Once you understand the basics of complex types, refer to the individual type topics when you need to refresh your memory about syntax
+      and examples:
+    </p>
+
+    <ul>
+      <li>
+        <xref href="impala_array.xml#array"/>
+      </li>
+
+      <li>
+        <xref href="impala_struct.xml#struct"/>
+      </li>
+
+      <li>
+        <xref href="impala_map.xml#map"/>
+      </li>
+    </ul>
+
+  </conbody>
+
+  <concept id="complex_types_benefits">
+
+    <title>Benefits of Impala Complex Types</title>
+
+    <conbody>
+
+      <p>
+        The reasons for using Impala complex types include the following:
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            You already have data produced by Hive or other non-Impala component that uses the complex type column names. You might need to
+            convert the underlying data to Parquet to use it with Impala.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Your data model originates with a non-SQL programming language or a NoSQL data management system. For example, if you are
+            representing Python data expressed as nested lists, dictionaries, and tuples, those data structures correspond closely to Impala
+            <codeph>ARRAY</codeph>, <codeph>MAP</codeph>, and <codeph>STRUCT</codeph> types.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Your analytic queries involving multiple tables could benefit from greater locality during join processing. By packing more
+            related data items within each HDFS data block, complex types let join queries avoid the network overhead of the traditional
+            Hadoop shuffle or broadcast join techniques.
+          </p>
+        </li>
+      </ul>
+
+      <p>
+        The Impala complex type support produces result sets with all scalar values, and the scalar components of complex types can be used
+        with all SQL clauses, such as <codeph>GROUP BY</codeph>, <codeph>ORDER BY</codeph>, all kinds of joins, subqueries, and inline
+        views. The ability to process complex type data entirely in SQL reduces the need to write application-specific code in Java or other
+        programming languages to deconstruct the underlying data structures.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="complex_types_overview">
+
+    <title>Overview of Impala Complex Types</title>
+
+    <conbody>
+
+      <p>
+<!--
+      Each <codeph>ARRAY</codeph>, <codeph>MAP</codeph>, or <codeph>STRUCT</codeph> column can include multiple instances of scalar types
+      such as <codeph>BIGINT</codeph> and <codeph>STRING</codeph>.
+-->
+        The <codeph>ARRAY</codeph> and <codeph>MAP</codeph> types are closely related: they represent collections with arbitrary numbers of
+        elements, where each element is the same type. In contrast, <codeph>STRUCT</codeph> groups together a fixed number of items into a
+        single element. The parts of a <codeph>STRUCT</codeph> element (the <term>fields</term>) can be of different types, and each field
+        has a name.
+      </p>
+
+      <p>
+        The elements of an <codeph>ARRAY</codeph> or <codeph>MAP</codeph>, or the fields of a <codeph>STRUCT</codeph>, can also be other
+        complex types. You can construct elaborate data structures with up to 100 levels of nesting. For example, you can make an
+        <codeph>ARRAY</codeph> whose elements are <codeph>STRUCT</codeph>s. Within each <codeph>STRUCT</codeph>, you can have some fields
+        that are <codeph>ARRAY</codeph>, <codeph>MAP</codeph>, or another kind of <codeph>STRUCT</codeph>. The Impala documentation uses the
+        terms complex and nested types interchangeably; for simplicity, it primarily uses the term complex types, to encompass all the
+        properties of these types.
+      </p>
+
+      <p>
+        When visualizing your data model in familiar SQL terms, you can think of each <codeph>ARRAY</codeph> or <codeph>MAP</codeph> as a
+        miniature table, and each <codeph>STRUCT</codeph> as a row within such a table. By default, the table represented by an
+        <codeph>ARRAY</codeph> has two columns, <codeph>POS</codeph> to represent ordering of elements, and <codeph>ITEM</codeph>
+        representing the value of each element. Likewise, by default, the table represented by a <codeph>MAP</codeph> encodes key-value
+        pairs, and therefore has two columns, <codeph>KEY</codeph> and <codeph>VALUE</codeph>.
+<!--
+        When you use a <codeph>STRUCT</codeph> as an
+        <codeph>ARRAY</codeph> element or the <codeph>VALUE</codeph> part of a <codeph>MAP</codeph>, the field names of the
+        <codeph>STRUCT</codeph> become additional columns in the result set.
+-->
+      </p>
+
+      <p>
+        The <codeph>ITEM</codeph> and <codeph>VALUE</codeph> names are only required for the very simplest kinds of <codeph>ARRAY</codeph>
+        and <codeph>MAP</codeph> columns, ones that hold only scalar values. When the elements within the <codeph>ARRAY</codeph> or
+        <codeph>MAP</codeph> are of type <codeph>STRUCT</codeph> rather than a scalar type, then the result set contains columns with names
+        corresponding to the <codeph>STRUCT</codeph> fields rather than <codeph>ITEM</codeph> or <codeph>VALUE</codeph>.
+      </p>
+
+<!--
+      <p>
+        <codeph>ARRAY</codeph> and <codeph>MAP</codeph> are both <term>collection</term> types, which can have a variable number of
+        elements; <codeph>ARRAY</codeph> and <codeph>MAP</codeph> are typically used as the top-level type of a table column.
+        <codeph>STRUCT</codeph> represents a single element and has a fixed number of fields; <codeph>STRUCT</codeph> is typically used as
+        the final, lowest level of a nested type definition.
+      </p>
+-->
+
+      <p>
+        You write most queries that process complex type columns using familiar join syntax, even though the data for both sides of the join
+        resides in a single table. The join notation brings together the scalar values from a row with the values from the complex type
+        columns for that same row. The final result set contains all scalar values, allowing you to do all the familiar filtering,
+        aggregation, ordering, and so on for the complex data entirely in SQL or using business intelligence tools that issue SQL queries.
+<!--
+        Instead of pulling together values from different tables, the join selects the specified values from both
+        the scalar columns, and from inside the complex type columns, producing a flattened result set consisting of all scalar values. When
+        doing a join query involving a complex type column, Impala derives the join key automatically, without the need to create additional
+        ID columns in the table.
+-->
+      </p>
+
+      <p>
+        Behind the scenes, Impala ensures that the processing for each row is done efficiently on a single host, without the network traffic
+        involved in broadcast or shuffle joins. The most common type of join query for tables with complex type columns is <codeph>INNER
+        JOIN</codeph>, which returns results only in those cases where the complex type contains some elements. Therefore, most query
+        examples in this section use either the <codeph>INNER JOIN</codeph> clause or the equivalent comma notation.
+      </p>
+
+      <note>
+        <p>
+          Although Impala can query complex types that are present in Parquet files, Impala currently cannot create new Parquet files
+          containing complex types. Therefore, the discussion and examples presume that you are working with existing Parquet data produced
+          through Hive, Spark, or some other source. See <xref href="#complex_types_ex_hive_etl"/> for examples of constructing Parquet data
+          files with complex type columns.
+        </p>
+
+        <p>
+          For learning purposes, you can create empty tables with complex type columns and practice query syntax, even if you do not have
+          sample data with the required structure.
+        </p>
+      </note>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="complex_types_design">
+
+    <title>Design Considerations for Complex Types</title>
+
+    <conbody>
+
+      <p>
+        When planning to use Impala complex types, and designing the Impala schema, first learn how this kind of schema differs from
+        traditional table layouts from the relational database and data warehousing fields. Because you might have already encountered
+        complex types in a Hadoop context while using Hive for ETL, also learn how to write high-performance analytic queries for complex
+        type data using Impala SQL syntax.
+      </p>
+
+      <p outputclass="toc inpage"/>
+
+    </conbody>
+
+    <concept id="complex_types_vs_rdbms">
+
+      <title>How Complex Types Differ from Traditional Data Warehouse Schemas</title>
+
+      <conbody>
+
+        <p>
+          Complex types let you associate arbitrary data structures with a particular row. If you are familiar with schema design for
+          relational database management systems or data warehouses, a schema with complex types has the following differences:
+        </p>
+
+        <ul>
+          <li>
+            <p>
+              Logically, related values can now be grouped tightly together in the same table.
+            </p>
+
+            <p>
+              In traditional data warehousing, related values were typically arranged in one of two ways:
+            </p>
+            <ul>
+              <li>
+                <p>
+                  Split across multiple normalized tables. Foreign key columns specified which rows from each table were associated with
+                  each other. This arrangement avoided duplicate data and therefore the data was compact, but join queries could be
+                  expensive because the related data had to be retrieved from separate locations. (In the case of distributed Hadoop
+                  queries, the joined tables might even be transmitted between different hosts in a cluster.)
+                </p>
+              </li>
+
+              <li>
+                <p>
+                  Flattened into a single denormalized table. Although this layout eliminated some potential performance issues by removing
+                  the need for join queries, the table typically became larger because values were repeated. The extra data volume could
+                  cause performance issues in other parts of the workflow, such as longer ETL cycles or more expensive full-table scans
+                  during queries.
+                </p>
+              </li>
+            </ul>
+            <p>
+              Complex types represent a middle ground that addresses these performance and volume concerns. By physically locating related
+              data within the same data files, complex types increase locality and reduce the expense of join queries. By associating an
+              arbitrary amount of data with a single row, complex types avoid the need to repeat lengthy values such as strings. Because
+              Impala knows which complex type values are associated with each row, you can save storage by avoiding artificial foreign key
+              values that are only used for joins. The flexibility of the <codeph>STRUCT</codeph>, <codeph>ARRAY</codeph>, and
+              <codeph>MAP</codeph> types lets you model familiar constructs such as fact and dimension tables from a data warehouse, and
+              wide tables representing sparse matrixes.
+            </p>
+          </li>
+        </ul>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="complex_types_physical">
+
+      <title>Physical Storage for Complex Types</title>
+
+      <conbody>
+
+        <p>
+          Physically, the scalar and complex columns in each row are located adjacent to each other in the same Parquet data file, ensuring
+          that they are processed on the same host rather than being broadcast across the network when cross-referenced within a query. This
+          co-location simplifies the process of copying, converting, and backing all the columns up at once. Because of the column-oriented
+          layout of Parquet files, you can still query only the scalar columns of a table without imposing the I/O penalty of reading the
+          (possibly large) values of the composite columns.
+        </p>
+
+        <p>
+          Within each Parquet data file, the constituent parts of complex type columns are stored in column-oriented format:
+        </p>
+
+        <ul>
+          <li>
+            <p>
+              Each field of a <codeph>STRUCT</codeph> type is stored like a column, with all the scalar values adjacent to each other and
+              encoded, compressed, and so on using the Parquet space-saving techniques.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              For an <codeph>ARRAY</codeph> containing scalar values, all those values (represented by the <codeph>ITEM</codeph>
+              pseudocolumn) are stored adjacent to each other.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              For a <codeph>MAP</codeph>, the values of the <codeph>KEY</codeph> pseudocolumn are stored adjacent to each other. If the
+              <codeph>VALUE</codeph> pseudocolumn is a scalar type, its values are also stored adjacent to each other.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              If an <codeph>ARRAY</codeph> element, <codeph>STRUCT</codeph> field, or <codeph>MAP</codeph> <codeph>VALUE</codeph> part is
+              another complex type, the column-oriented storage applies to the next level down (or the next level after that, and so on for
+              deeply nested types) where the final elements, fields, or values are of scalar types.
+            </p>
+          </li>
+        </ul>
+
+        <p>
+          The numbers represented by the <codeph>POS</codeph> pseudocolumn of an <codeph>ARRAY</codeph> are not physically stored in the
+          data files. They are synthesized at query time based on the order of the <codeph>ARRAY</codeph> elements associated with each row.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="complex_types_file_formats">
+
+      <title>File Format Support for Impala Complex Types</title>
+
+      <conbody>
+
+        <p>
+          Currently, Impala queries support complex type data only in the Parquet file format. See <xref href="impala_parquet.xml#parquet"/>
+          for details about the performance benefits and physical layout of this file format.
+        </p>
+
+        <p>
+          Each table, or each partition within a table, can have a separate file format, and you can change file format at the table or
+          partition level through an <codeph>ALTER TABLE</codeph> statement. Because this flexibility makes it difficult to guarantee ahead
+          of time that all the data files for a table or partition are in a compatible format, Impala does not throw any errors when you
+          change the file format for a table or partition using <codeph>ALTER TABLE</codeph>. Any errors come at runtime when Impala
+          actually processes a table or partition that contains nested types and is not in one of the supported formats. If a query on a
+          partitioned table only processes some partitions, and all those partitions are in one of the supported formats, the query
+          succeeds.
+        </p>
+
+        <p>
+          Because Impala does not parse the data structures containing nested types for unsupported formats such as text, Avro,
+          SequenceFile, or RCFile, you cannot use data files in these formats with Impala, even if the query does not refer to the nested
+          type columns. Also, if a table using an unsupported format originally contained nested type columns, and then those columns were
+          dropped from the table using <codeph>ALTER TABLE ... DROP COLUMN</codeph>, any existing data files in the table still contain the
+          nested type data and Impala queries on that table will generate errors.
+        </p>
+
+        <note rev="2.6.0 IMPALA-2844">
+          <p rev="2.6.0 IMPALA-2844">
+            The one exception to the preceding rule is <codeph>COUNT(*)</codeph> queries on RCFile tables that include complex types.
+            Such queries are allowed in <keyword keyref="impala26_full"/> and higher.
+          </p>
+        </note>
+
+        <p>
+          You can perform DDL operations (even <codeph>CREATE TABLE</codeph>) for tables involving complex types in file formats other than
+          Parquet. The DDL support lets you set up intermediate tables in your ETL pipeline, to be populated by Hive, before the final stage
+          where the data resides in a Parquet table and is queryable by Impala. Also, you can have a partitioned table with complex type
+          columns that uses a non-Parquet format, and use <codeph>ALTER TABLE</codeph> to change the file format to Parquet for individual
+          partitions. When you put Parquet data files into those partitions, Impala can execute queries against that data as long as the
+          query does not involve any of the non-Parquet partitions.
+        </p>
+
+        <p>
+          If you use the <cmdname>parquet-tools</cmdname> command to examine the structure of a Parquet data file that includes complex
+          types, you see that both <codeph>ARRAY</codeph> and <codeph>MAP</codeph> are represented as a <codeph>Bag</codeph> in Parquet
+          terminology, with all fields marked <codeph>Optional</codeph> because Impala allows any column to be nullable.
+        </p>
+
+        <p>
+          Impala supports either 2-level and 3-level encoding within each Parquet data file. When constructing Parquet data files outside
+          Impala, use either encoding style but do not mix 2-level and 3-level encoding within the same data file.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="complex_types_vs_normalization">
+
+      <title>Choosing Between Complex Types and Normalized Tables</title>
+
+      <conbody>
+
+        <p>
+          Choosing between multiple normalized fact and dimension tables, or a single table containing complex types, is an important design
+          decision.
+        </p>
+
+        <ul>
+          <li>
+            <p>
+              If you are coming from a traditional database or data warehousing background, you might be familiar with how to split up data
+              between tables. Your business intelligence tools might already be optimized for dealing with this kind of multi-table scenario
+              through join queries.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              If you are pulling data from Impala into an application written in a programming language that has data structures analogous
+              to the complex types, such as Python or Java, complex types in Impala could simplify data interchange and improve
+              understandability and reliability of your program logic.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              You might already be faced with existing infrastructure or receive high volumes of data that assume one layout or the other.
+              For example, complex types are popular with web-oriented applications, for example to keep information about an online user
+              all in one place for convenient lookup and analysis, or to deal with sparse or constantly evolving data fields.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              If some parts of the data change over time while related data remains constant, using multiple normalized tables lets you
+              replace certain parts of the data without reloading the entire data set. Conversely, if you receive related data all bundled
+              together, such as in JSON files, using complex types can save the overhead of splitting the related items across multiple
+              tables.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              From a performance perspective:
+            </p>
+            <ul>
+              <li>
+                <p>
+                  In Parquet tables, Impala can skip columns that are not referenced in a query, avoiding the I/O penalty of reading the
+                  embedded data. When complex types are nested within a column, the data is physically divided at a very granular level; for
+                  example, a query referring to data nested multiple levels deep in a complex type column does not have to read all the data
+                  from that column, only the data for the relevant parts of the column type hierarchy.
+<!-- Avro not supported in 5.5 / 2.3: Avro tables might experience some performance overhead due to
+                the need to skip past the complex type columns in each row when reading the data. -->
+                </p>
+              </li>
+
+              <li>
+                <p>
+                  Complex types avoid the possibility of expensive join queries when data from fact and dimension tables is processed in
+                  parallel across multiple hosts. All the information for a row containing complex types is typically to be in the same data
+                  block, and therefore does not need to be transmitted across the network when joining fields that are all part of the same
+                  row.
+                </p>
+              </li>
+
+              <li>
+                <p>
+                  The tradeoff with complex types is that fewer rows fit in each data block. Whether it is better to have more data blocks
+                  with fewer rows, or fewer data blocks with many rows, depends on the distribution of your data and the characteristics of
+                  your query workload. If the complex columns are rarely referenced, using them might lower efficiency. If you are seeing
+                  low parallelism due to a small volume of data (relatively few data blocks) in each table partition, increasing the row
+                  size by including complex columns might produce more data blocks and thus spread the work more evenly across the cluster.
+                  See <xref href="impala_scalability.xml#scalability"/> for more on this advanced topic.
+                </p>
+              </li>
+            </ul>
+          </li>
+        </ul>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="complex_types_hive">
+
+      <title>Differences Between Impala and Hive Complex Types</title>
+
+      <conbody>
+
+<!-- HiveQL functions like nested type constructors and posexplode(): https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF -->
+
+<!-- HiveQL complex types: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-ComplexTypes -->
+
+<!-- HiveQL lateral views: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LateralView -->
+
+        <p>
+          Impala can query Parquet tables containing <codeph>ARRAY</codeph>, <codeph>STRUCT</codeph>, and <codeph>MAP</codeph> columns
+          produced by Hive. There are some differences to be aware of between the Impala SQL and HiveQL syntax for complex types, primarily
+          for queries.
+        </p>
+
+        <p>
+          The syntax for specifying <codeph>ARRAY</codeph>, <codeph>STRUCT</codeph>, and <codeph>MAP</codeph> types in a <codeph>CREATE
+          TABLE</codeph> statement is compatible between Impala and Hive.
+        </p>
+
+        <p>
+          Because Impala <codeph>STRUCT</codeph> columns include user-specified field names, you use the <codeph>NAMED_STRUCT()</codeph>
+          constructor in Hive rather than the <codeph>STRUCT()</codeph> constructor when you populate an Impala <codeph>STRUCT</codeph>
+          column using a Hive <codeph>INSERT</codeph> statement.
+        </p>
+
+        <p>
+          The Hive <codeph>UNION</codeph> type is not currently supported in Impala.
+        </p>
+
+        <p>
+          While Impala usually aims for a high degree of compatibility with HiveQL query syntax, Impala syntax differs from Hive for queries
+          involving complex types. The differences are intended to provide extra flexibility for queries involving these kinds of tables.
+        </p>
+
+        <ul>
+          <li>
+            Impala uses dot notation for referring to element names or elements within complex types, and join notation for
+            cross-referencing scalar columns with the elements of complex types within the same row, rather than the <codeph>LATERAL
+            VIEW</codeph> clause and <codeph>EXPLODE()</codeph> function of HiveQL.
+          </li>
+
+          <li>
+            Using join notation lets you use all the kinds of join queries with complex type columns. For example, you can use a
+            <codeph>LEFT OUTER JOIN</codeph>, <codeph>LEFT ANTI JOIN</codeph>, or <codeph>LEFT SEMI JOIN</codeph> query to evaluate
+            different scenarios where the complex columns do or do not contain any elements.
+          </li>
+
+          <li>
+            You can include references to collection types inside subqueries and inline views. For example, you can construct a
+            <codeph>FROM</codeph> clause where one of the <q>tables</q> is a subquery against a complex type column, or use a subquery
+            against a complex type column as the argument to an <codeph>IN</codeph> or <codeph>EXISTS</codeph> clause.
+          </li>
+
+          <li>
+            The Impala pseudocolumn <codeph>POS</codeph> lets you retrieve the position of elements in an array along with the elements
+            themselves, equivalent to the <codeph>POSEXPLODE()</codeph> function of HiveQL. You do not use index notation to retrieve a
+            single array element in a query; the join query loops through the array elements and you use <codeph>WHERE</codeph> clauses to
+            specify which elements to return.
+          </li>
+
+          <li>
+            <p>
+              Join clauses involving complex type columns do not require an <codeph>ON</codeph> or <codeph>USING</codeph> clause. Impala
+              implicitly applies the join key so that the correct array entries or map elements are associated with the correct row from the
+              table.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              Impala does not currently support the <codeph>UNION</codeph> complex type.
+            </p>
+          </li>
+        </ul>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="complex_types_limits">
+
+      <title>Limitations and Restrictions for Complex Types</title>
+
+      <conbody>
+
+        <p>
+          Complex type columns can only be used in tables or partitions with the Parquet file format.
+        </p>
+
+        <p>
+          Complex type columns cannot be used as partition key columns in a partitioned table.
+        </p>
+
+        <p>
+          When you use complex types with the <codeph>ORDER BY</codeph>, <codeph>GROUP BY</codeph>, <codeph>HAVING</codeph>, or
+          <codeph>WHERE</codeph> clauses, you cannot refer to the column name by itself. Instead, you refer to the names of the scalar
+          values within the complex type, such as the <codeph>ITEM</codeph>, <codeph>POS</codeph>, <codeph>KEY</codeph>, or
+          <codeph>VALUE</codeph> pseudocolumns, or the field names from a <codeph>STRUCT</codeph>.
+        </p>
+
+        <p>
+          The maximum depth of nesting for complex types is 100 levels.
+        </p>
+
+        <p conref="../shared/impala_common.xml#common/complex_types_max_length"/>
+
+        <p>
+          For ideal performance and scalability, use small or medium-sized collections, where all the complex columns contain at most a few
+          hundred megabytes per row. Remember, all the columns of a row are stored in the same HDFS data block, whose size in Parquet files
+          typically ranges from 256 MB to 1 GB.
+        </p>
+
+        <p>
+          Including complex type columns in a table introduces some overhead that might make queries that do not reference those columns
+          somewhat slower than Impala queries against tables without any complex type columns. Expect at most a 2x slowdown compared to
+          tables that do not have any complex type columns.
+        </p>
+
+        <p>
+          Currently, the <codeph>COMPUTE STATS</codeph> statement does not collect any statistics for columns containing complex types.
+          Impala uses heuristics to construct execution plans involving complex type columns.
+        </p>
+
+        <p>
+          Currently, Impala built-in functions and user-defined functions cannot accept complex types as parameters or produce them as
+          function return values. (When the complex type values are materialized in an Impala result set, the result set contains the scalar
+          components of the values, such as the <codeph>POS</codeph> or <codeph>ITEM</codeph> for an <codeph>ARRAY</codeph>, the
+          <codeph>KEY</codeph> or <codeph>VALUE</codeph> for a <codeph>MAP</codeph>, or the fields of a <codeph>STRUCT</codeph>; these
+          scalar data items <i>can</i> be used with built-in functions and UDFs as usual.)
+        </p>
+
+        <p conref="../shared/impala_common.xml#common/complex_types_read_only"/>
+
+        <p>
+          Currently, Impala can query complex type columns only from Parquet tables or Parquet partitions within partitioned tables.
+          Although you can use complex types in tables with Avro, text, and other file formats as part of your ETL pipeline, for example as
+          intermediate tables populated through Hive, doing analytics through Impala requires that the data eventually ends up in a Parquet
+          table. The requirement for Parquet data files means that you can use complex types with Impala tables hosted on other kinds of
+          file storage systems such as Isilon and Amazon S3, but you cannot use Impala to query complex types from HBase tables. See
+          <xref href="impala_complex_types.xml#complex_types_file_formats"/> for more details.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+  </concept>
+
+  <concept id="complex_types_using">
+
+    <title>Using Complex Types from SQL</title>
+
+    <conbody>
+
+      <p>
+        When using complex types through SQL in Impala, you learn the notation for <codeph>&lt; &gt;</codeph> delimiters for the complex
+        type columns in <codeph>CREATE TABLE</codeph> statements, and how to construct join queries to <q>unpack</q> the scalar values
+        nested inside the complex data structures. You might need to condense a traditional RDBMS or data warehouse schema into a smaller
+        number of Parquet tables, and use Hive, Spark, Pig, or other mechanism outside Impala to populate the tables with data.
+      </p>
+
+      <p outputclass="toc inpage"/>
+
+    </conbody>
+
+    <concept id="nested_types_ddl">
+
+      <title>Complex Type Syntax for DDL Statements</title>
+
+      <conbody>
+
+        <p>
+          The definition of <varname>data_type</varname>, as seen in the <codeph>CREATE TABLE</codeph> and <codeph>ALTER TABLE</codeph>
+          statements, now includes complex types in addition to primitive types:
+        </p>
+
+<codeblock>  primitive_type
+| array_type
+| map_type
+| struct_type
+</codeblock>
+
+        <p>
+          Unions are not currently supported.
+        </p>
+
+        <p>
+          Array, struct, and map column type declarations are specified in the <codeph>CREATE TABLE</codeph> statement. You can also add or
+          change the type of complex columns through the <codeph>ALTER TABLE</codeph> statement.
+        </p>
+
+        <note>
+          <p>
+            Currently, Impala queries allow complex types only in tables that use the Parquet format. If an Impala query encounters complex
+            types in a table or partition using another file format, the query returns a runtime error.
+          </p>
+
+          <p>
+            The Impala DDL support for complex types works for all file formats, so that you can create tables using text or other
+            non-Parquet formats for Hive to use as staging tables in an ETL cycle that ends with the data in a Parquet table. You can also
+            use <codeph>ALTER TABLE ... SET FILEFORMAT PARQUET</codeph> to change the file format of an existing table containing complex
+            types to Parquet, after which Impala can query it. Make sure to load Parquet files into the table after changing the file
+            format, because the <codeph>ALTER TABLE ... SET FILEFORMAT</codeph> statement does not convert existing data to the new file
+            format.
+          </p>
+        </note>
+
+        <p conref="../shared/impala_common.xml#common/complex_types_partitioning"/>
+
+        <p>
+          Because use cases for Impala complex types require that you already have Parquet data files produced outside of Impala, you can
+          use the Impala <codeph>CREATE TABLE LIKE PARQUET</codeph> syntax to produce a table with columns that match the structure of an
+          existing Parquet file, including complex type columns for nested data structures. Remember to include the <codeph>STORED AS
+          PARQUET</codeph> clause in this case, because even with <codeph>CREATE TABLE LIKE PARQUET</codeph>, the default file format of the
+          resulting table is still text.
+        </p>
+
+        <p>
+          Because the complex columns are omitted from the result set of an Impala <codeph>SELECT *</codeph> or <codeph>SELECT
+          <varname>col_name</varname></codeph> query, and because Impala currently does not support writing Parquet files with complex type
+          columns, you cannot use the <codeph>CREATE TABLE AS SELECT</codeph> syntax to create a table with nested type columns.
+        </p>
+
+        <note>
+          <p>
+            Once you have a table set up with complex type columns, use the <codeph>DESCRIBE</codeph> and <codeph>SHOW CREATE TABLE</codeph>
+            statements to see the correct notation with <codeph>&lt;</codeph> and <codeph>&gt;</codeph> delimiters and comma and colon
+            separators within the complex type definitions. If you do not have existing data with the same layout as the table, you can
+            query the empty table to practice with the notation for the <codeph>SELECT</codeph> statement. In the <codeph>SELECT</codeph>
+            list, you use dot notation and pseudocolumns such as <codeph>ITEM</codeph>, <codeph>KEY</codeph>, and <codeph>VALUE</codeph> for
+            referring to items within the complex type columns. In the <codeph>FROM</codeph> clause, you use join notation to construct
+            table aliases for any referenced <codeph>ARRAY</codeph> and <codeph>MAP</codeph> columns.
+          </p>
+        </note>
+
+<!-- To do: show some simple CREATE TABLE statements for each of the complex types, without so much backstory for the schema. -->
+
+        <p>
+          For example, when defining a table that holds contact information, you might represent phone numbers differently depending on the
+          expected layout and relationships of the data, and how well you can predict those properties in advance.
+        </p>
+
+        <p>
+          Here are different ways that you might represent phone numbers in a traditional relational schema, with equivalent representations
+          using complex types.
+        </p>
+
+        <fig id="complex_types_phones_flat_fixed">
+
+          <title>Traditional Relational Representation of Phone Numbers: Single Table</title>
+
+          <p>
+            The traditional, simplest way to represent phone numbers in a relational table is to store all contact info in a single table,
+            with all columns having scalar types, and each potential phone number represented as a separate column. In this example, each
+            person can only have these 3 types of phone numbers. If the person does not have a particular kind of phone number, the
+            corresponding column is <codeph>NULL</codeph> for that row.
+          </p>
+
+<codeblock>
+CREATE TABLE contacts_fixed_phones
+(
+    id BIGINT
+  , name STRING
+  , address STRING
+  , home_phone STRING
+  , work_phone STRING
+  , mobile_phone STRING
+) STORED AS PARQUET;
+</codeblock>
+
+        </fig>
+
+        <fig id="complex_types_phones_array">
+
+          <title>An Array of Phone Numbers</title>
+
+          <p>
+            Using a complex type column to represent the phone numbers adds some extra flexibility. Now there could be an unlimited number
+            of phone numbers. Because the array elements have an order but not symbolic names, you could decide in advance that
+            phone_number[0] is the home number, [1] is the work number, [2] is the mobile number, and so on. (In subsequent examples, you
+            will see how to create a more flexible naming scheme using other complex type variations, such as a <codeph>MAP</codeph> or an
+            <codeph>ARRAY</codeph> where each element is a <codeph>STRUCT</codeph>.)
+          </p>
+
+<codeblock><![CDATA[
+CREATE TABLE contacts_array_of_phones
+(
+    id BIGINT
+  , name STRING
+  , address STRING
+  , phone_number ARRAY < STRING >
+) STORED AS PARQUET;
+]]>
+</codeblock>
+
+        </fig>
+
+        <fig id="complex_types_phones_map">
+
+          <title>A Map of Phone Numbers</title>
+
+          <p>
+            Another way to represent an arbitrary set of phone numbers is with a <codeph>MAP</codeph> column. With a <codeph>MAP</codeph>,
+            each element is associated with a key value that you specify, which could be a numeric, string, or other scalar type. This
+            example uses a <codeph>STRING</codeph> key to give each phone number a name, such as <codeph>'home'</codeph> or
+            <codeph>'mobile'</codeph>. A query could filter the data based on the key values, or display the key values in reports.
+          </p>
+
+<codeblock><![CDATA[
+CREATE TABLE contacts_unlimited_phones
+(
+  id BIGINT, name STRING, address STRING, phone_number MAP < STRING,STRING >
+) STORED AS PARQUET;
+]]>
+</codeblock>
+
+        </fig>
+
+        <fig id="complex_types_phones_flat_normalized">
+
+          <title>Traditional Relational Representation of Phone Numbers: Normalized Tables</title>
+
+          <p>
+            If you are an experienced database designer, you already know how to work around the limitations of the single-table schema from
+            <xref href="#nested_types_ddl/complex_types_phones_flat_fixed"/>. By normalizing the schema, with the phone numbers in their own
+            table, you can associate an arbitrary set of phone numbers with each person, and associate additional details with each phone
+            number, such as whether it is a home, work, or mobile phone.
+          </p>
+
+          <p>
+            The flexibility of this approach comes with some drawbacks. Reconstructing all the data for a particular person requires a join
+            query, which might require performance tuning on Hadoop because the data from each table might be transmitted from a different
+            host. Data management tasks such as backups and refreshing the data require dealing with multiple tables instead of a single
+            table.
+          </p>
+
+          <p>
+            This example illustrates a traditional database schema to store contact info normalized across 2 tables. The fact table
+            establishes the identity and basic information about person. A dimension table stores information only about phone numbers,
+            using an ID value to associate each phone number with a person ID from the fact table. Each person can have 0, 1, or many
+            phones; the categories are not restricted to a few predefined ones; and the phone table can contain as many columns as desired,
+            to represent all sorts of details about each phone number.
+          </p>
+
+<codeblock>
+CREATE TABLE fact_contacts (id BIGINT, name STRING, address STRING) STORED AS PARQUET;
+CREATE TABLE dim_phones
+(
+    contact_id BIGINT
+  , category STRING
+  , international_code STRING
+  , area_code STRING
+  , exchange STRING
+  , extension STRING
+  , mobile BOOLEAN
+  , carrier STRING
+  , current BOOLEAN
+  , service_start_date TIMESTAMP
+  , service_end_date TIMESTAMP
+)
+STORED AS PARQUET;
+</codeblock>
+
+        </fig>
+
+        <fig id="complex_types_phones_array_struct">
+
+          <title>Phone Numbers Represented as an Array of Structs</title>
+
+          <p>
+            To represent a schema equivalent to the one from <xref href="#nested_types_ddl/complex_types_phones_flat_normalized"/> using
+            complex types, this example uses an <codeph>ARRAY</codeph> where each array element is a <codeph>STRUCT</codeph>. As with the
+            earlier complex type examples, each person can have an arbitrary set of associated phone numbers. Making each array element into
+            a <codeph>STRUCT</codeph> lets us associate multiple data items with each phone number, and give a separate name and type to
+            each data item. The <codeph>STRUCT</codeph> fields of the <codeph>ARRAY</codeph> elements reproduce the columns of the dimension
+            table from the previous example.
+          </p>
+
+          <p>
+            You can do all the same kinds of queries with the complex type schema as with the normalized schema from the previous example.
+            The advantages of the complex type design are in the areas of convenience and performance. Now your backup and ETL processes
+            only deal with a single table. When a query uses a join to cross-reference the information about a person with their associated
+            phone numbers, all the relevant data for each row resides in the same HDFS data block, meaning each row can be processed on a
+            single host without requiring network transmission.
+          </p>
+
+<codeblock><![CDATA[
+CREATE TABLE contacts_detailed_phones
+(
+  id BIGINT, name STRING, address STRING
+    , phone ARRAY < STRUCT <
+        category: STRING
+      , international_code: STRING
+      , area_code: STRING
+      , exchange: STRING
+      , extension: STRING
+      , mobile: BOOLEAN
+      , carrier: STRING
+      , current: BOOLEAN
+      , service_start_date: TIMESTAMP
+      , service_end_date: TIMESTAMP
+    >>
+) STORED AS PARQUET;
+]]>
+</codeblock>
+
+        </fig>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="complex_types_sql">
+
+      <title>SQL Statements that Support Complex Types</title>
+
+      <conbody>
+
+        <p>
+          The Impala SQL statements that support complex types are currently
+          <codeph><xref href="impala_create_table.xml#create_table">CREATE TABLE</xref></codeph>,
+          <codeph><xref href="impala_alter_table.xml#alter_table">ALTER TABLE</xref></codeph>,
+          <codeph><xref href="impala_describe.xml#describe">DESCRIBE</xref></codeph>,
+          <codeph><xref href="impala_load_data.xml#load_data">LOAD DATA</xref></codeph>, and
+          <codeph><xref href="impala_select.xml#select">SELECT</xref></codeph>. That is, currently Impala can create or alter tables
+          containing complex type columns, examine the structure of a table containing complex type columns, import existing data files
+          containing complex type columns into a table, and query Parquet tables containing complex types.
+        </p>
+
+        <p conref="../shared/impala_common.xml#common/complex_types_read_only"/>
+
+        <p outputclass="toc inpage"/>
+
+      </conbody>
+
+      <concept id="complex_types_ddl">
+
+        <title>DDL Statements and Complex Types</title>
+
+        <conbody>
+
+          <p>
+            Column specifications for complex or nested types use <codeph>&lt;</codeph> and <codeph>&gt;</codeph> delimiters:
+          </p>
+
+<codeblock><![CDATA[-- What goes inside the < > for an ARRAY is a single type, either a scalar or another
+-- complex type (ARRAY, STRUCT, or MAP).
+CREATE TABLE array_t
+(
+  id BIGINT,
+  a1 ARRAY <STRING>,
+  a2 ARRAY <BIGINT>,
+  a3 ARRAY <TIMESTAMP>,
+  a4 ARRAY <STRUCT <f1: STRING, f2: INT, f3: BOOLEAN>>
+)
+STORED AS PARQUET;
+
+-- What goes inside the < > for a MAP is two comma-separated types specifying the types of the key-value pair:
+-- a scalar type representing the key, and a scalar or complex type representing the value.
+CREATE TABLE map_t
+(
+  id BIGINT,
+  m1 MAP <STRING, STRING>,
+  m2 MAP <STRING, BIGINT>,
+  m3 MAP <BIGINT, STRING>,
+  m4 MAP <BIGINT, BIGINT>,
+  m5 MAP <STRING, ARRAY <STRING>>
+)
+STORED AS PARQUET;
+
+-- What goes inside the < > for a STRUCT is a comma-separated list of fields, each field defined as
+-- name:type. The type can be a scalar or a complex type. The field names for each STRUCT do not clash
+-- with the names of table columns or fields in other STRUCTs. A STRUCT is most often used inside
+-- an ARRAY or a MAP rather than as a top-level column.
+CREATE TABLE struct_t
+(
+  id BIGINT,
+  s1 STRUCT <f1: STRING, f2: BIGINT>,
+  s2 ARRAY <STRUCT <f1: INT, f2: TIMESTAMP>>,
+  s3 MAP <BIGINT, STRUCT <name: STRING, birthday: TIMESTAMP>>
+)
+STORED AS PARQUET;
+]]>
+</codeblock>
+
+        </conbody>
+
+      </concept>
+
+      <concept id="complex_types_queries">
+
+        <title>Queries and Complex Types</title>
+
+        <conbody>
+
+<!-- Hive does the JSON output business: http://www.datascience-labs.com/hive/hiveql-data-manipulation/ -->
+
+<!-- SELECT * works but skips any nested type coloumns. -->
+
+          <p>
+            The result set of an Impala query always contains all scalar types; the elements and fields within any complex type queries must
+            be <q>unpacked</q> using join queries. A query cannot directly retrieve the entire value for a complex type column. Impala
+            returns an error in this case. Queries using <codeph>SELECT *</codeph> are allowed for tables with complex types, but the
+            columns with complex types are skipped.
+          </p>
+
+          <p>
+            The following example shows how referring directly to a complex type column returns an error, while <codeph>SELECT *</codeph> on
+            the same table succeeds, but only retrieves the scalar columns.
+          </p>
+
+          <note conref="../shared/impala_common.xml#common/complex_type_schema_pointer"/>
+
+<!-- Original error message:
+ERROR: AnalysisException: Expr 'c_orders' in select list returns a complex type 'ARRAY<STRUCT<o_orderkey:BIGINT,o_orderstatus:STRING,o_totalprice:DECIMAL(12,2),o_orderdate:STRING,o_orderpriority:STRING,o_clerk:STRING,o_shippriority:INT,o_comment:STRING,o_lineitems:ARRAY<STRUCT<l_partkey:BIGINT,l_suppkey:BIGINT,l_linenumber:INT,l_quantity:DECIMAL(12,2),l_extendedprice:DECIMAL(12,2),l_discount:DECIMAL(12,2),l_tax:DECIMAL(12,2),l_returnflag:STRING,l_linestatus:STRING,l_shipdate:STRING,l_commitdate:STRING,l_receiptdate:STRING,l_shipinstruct:STRING,l_shipmode:STRING,l_comment:STRING>>>>'.
+-->
+
+<codeblock><![CDATA[SELECT c_orders FROM customer LIMIT 1;
+ERROR: AnalysisException: Expr 'c_orders' in select list returns a complex type 'ARRAY<STRUCT<o_orderkey:BIGINT,o_orderstatus:STRING, ... l_receiptdate:STRING,l_shipinstruct:STRING,l_shipmode:STRING,l_comment:STRING>>>>'.
+Only scalar types are allowed in the select list.
+
+-- Original column has several scalar and one complex column.
+DESCRIBE customer;
++--------------+------------------------------------+
+| name         | type                               |
++--------------+------------------------------------+
+| c_custkey    | bigint                             |
+| c_name       | string                             |
+...
+| c_orders     | array<struct<                      |
+|              |   o_orderkey:bigint,               |
+|              |   o_orderstatus:string,            |
+|              |   o_totalprice:decimal(12,2),      |
+...
+|              | >>                                 |
++--------------+------------------------------------+
+
+-- When we SELECT * from that table, only the scalar columns come back in the result set.
+CREATE TABLE select_star_customer STORED AS PARQUET AS SELECT * FROM customer;
++------------------------+
+| summary                |
++------------------------+
+| Inserted 150000 row(s) |
++------------------------+
+
+-- The c_orders column, being of complex type, was not included in the SELECT * result set.
+DESC select_star_customer;
++--------------+---------------+
+| name         | type          |
++--------------+---------------+
+| c_custkey    | bigint        |
+| c_name       | string        |
+| c_address    | string        |
+| c_nationkey  | smallint      |
+| c_phone      | string        |
+| c_acctbal    | decimal(12,2) |
+| c_mktsegment | string        |
+| c_comment    | string        |
++--------------+---------------+
+]]>
+</codeblock>
+
+<!-- To do: These "references to..." bits could be promoted to their own 'expressions' subheads. -->
+
+          <p>
+            References to fields within <codeph>STRUCT</codeph> columns use dot notation. If the field name is unambiguous, you can omit
+            qualifiers such as table name, column name, or even the <codeph>ITEM</codeph> or <codeph>VALUE</codeph> pseudocolumn names for
+            <codeph>STRUCT</codeph> elements inside an <codeph>ARRAY</codeph> or a <codeph>MAP</codeph>.
+          </p>
+
+<!-- To do: rewrite example to use CUSTOMER table. -->
+
+<!-- Don't think TPC-H schema has a bare STRUCT to use in such a simple query though. -->
+
+<!-- Perhaps reuse the STRUCT_DEMO example here. -->
+
+<codeblock>SELECT id, address.city FROM customers WHERE address.zip = 94305;
+</codeblock>
+
+          <p>
+            References to elements within <codeph>ARRAY</codeph> columns use the <codeph>ITEM</codeph> pseudocolumn:
+          </p>
+
+<!-- To do: shorten qualified names. -->
+
+<codeblock>select r_name, r_nations.item.n_name from region, region.r_nations limit 7;
++--------+----------------+
+| r_name | item.n_name    |
++--------+----------------+
+| EUROPE | UNITED KINGDOM |
+| EUROPE | RUSSIA         |
+| EUROPE | ROMANIA        |
+| EUROPE | GERMANY        |
+| EUROPE | FRANCE         |
+| ASIA   | VIETNAM        |
+| ASIA   | CHINA          |
++--------+----------------+
+</codeblock>
+
+          <p>
+            References to fields within <codeph>MAP</codeph> columns use the <codeph>KEY</codeph> and <codeph>VALUE</codeph> pseudocolumns.
+            In this example, once the query establishes the alias <codeph>MAP_FIELD</codeph> for a <codeph>MAP</codeph> column with a
+            <codeph>STRING</codeph> key and an <codeph>INT</codeph> value, the query can refer to <codeph>MAP_FIELD.KEY</codeph> and
+            <codeph>MAP_FIELD.VALUE</codeph>, which have zero, one, or many instances for each row from the containing table.
+          </p>
+
+<codeblock><![CDATA[DESCRIBE table_0;
++---------+-----------------------+
+| name    | type                  |
++---------+-----------------------+
+| field_0 | string                |
+| field_1 | map<string,int>       |
+...
+
+SELECT field_0, map_field.key, map_field.value
+  FROM table_0, table_0.field_1 AS map_field
+WHERE length(field_0) = 1
+LIMIT 10;
++---------+-----------+-------+
+| field_0 | key       | value |
++---------+-----------+-------+
+| b       | gshsgkvd  | NULL  |
+| b       | twrtcxj6  | 18    |
+| b       | 2vp5      | 39    |
+| b       | fh0s      | 13    |
+| v       | 2         | 41    |
+| v       | 8b58mz    | 20    |
+| v       | hw        | 16    |
+| v       | 65l388pyt | 29    |
+| v       | 03k68g91z | 30    |
+| v       | r2hlg5b   | NULL  |
++---------+-----------+-------+
+]]>
+</codeblock>
+
+<!-- To do: refer to or reuse examples from the other subtopics that discuss pseudocolumns etc. -->
+
+          <p>
+            When complex types are nested inside each other, you use a combination of joins, pseudocolumn names, and dot notation to refer
+            to specific fields at the appropriate level. This is the most frequent form of query syntax for complex columns, because the
+            typical use case involves two levels of complex types, such as an <codeph>ARRAY</codeph> of <codeph>STRUCT</codeph> elements.
+          </p>
+
+<!-- To do: rewrite example to use CUSTOMER table. -->
+
+<!-- This is my own manufactured example so I have the table, and the query works, but I don't have sample data to show. -->
+
+<codeblock>SELECT id, phone_numbers.area_code FROM contact_info_many_structs INNER JOIN contact_info_many_structs.phone_numbers phone_numbers LIMIT 3;
+</codeblock>
+
+          <p>
+            You can express relationships between <codeph>ARRAY</codeph> and <codeph>MAP</codeph> columns at different levels as joins. You
+            include comparison operators between fields at the top level and within the nested type columns so that Impala can do the
+            appropriate join operation.
+          </p>
+
+<!-- Don't think TPC-H schema has any instances where outer field matches up with inner one though. -->
+
+<!-- But don't think this usage is important enough to call out at this early point. Hide the example for now. -->
+
+<!--
+<codeblock>SELECT o.txn_id FROM customers c, c.orders o WHERE o.cc = c.preferred_cc;
+SELECT c.id, o.txn_id FROM customers c, c.orders o;
+</codeblock>
+-->
+
+<!-- To do: move these examples down, to the examples subtopic at the end. -->
+
+          <note conref="../shared/impala_common.xml#common/complex_type_schema_pointer"/>
+
+          <p>
+            For example, the following queries work equivalently. They each return customer and order data for customers that have at least
+            one order.
+          </p>
+
+<codeblock>SELECT c.c_name, o.o_orderkey FROM customer c, c.c_orders o LIMIT 5;
++--------------------+------------+
+| c_name             | o_orderkey |
++--------------------+------------+
+| Customer#000072578 | 558821     |
+| Customer#000072578 | 2079810    |
+| Customer#000072578 | 5768068    |
+| Customer#000072578 | 1805604    |
+| Customer#000072578 | 3436389    |
++--------------------+------------+
+
+SELECT c.c_name, o.o_orderkey FROM customer c INNER JOIN c.c_orders o LIMIT 5;
++--------------------+------------+
+| c_name             | o_orderkey |
++--------------------+------------+
+| Customer#000072578 | 558821     |
+| Customer#000072578 | 2079810    |
+| Customer#000072578 | 5768068    |
+| Customer#000072578 | 1805604    |
+| Customer#000072578 | 3436389    |
++--------------------+------------+
+</codeblock>
+
+          <p>
+            The following query using an outer join returns customers that have orders, plus customers with no orders (no entries in the
+            <codeph>C_ORDERS</codeph> array):
+          </p>
+
+<codeblock><![CDATA[SELECT c.c_custkey, o.o_orderkey
+  FROM customer c LEFT OUTER JOIN c.c_orders o
+LIMIT 5;
++-----------+------------+
+| c_custkey | o_orderkey |
++-----------+------------+
+| 60210     | NULL       |
+| 147873    | NULL       |
+| 72578     | 558821     |
+| 72578     | 2079810    |
+| 72578     | 5768068    |
++-----------+------------+
+]]>
+</codeblock>
+
+          <p>
+            The following query returns <i>only</i> customers that have no orders. (With <codeph>LEFT ANTI JOIN</codeph> or <codeph>LEFT
+            SEMI JOIN</codeph>, the query can only refer to columns from the left-hand table, because by definition there is no matching
+            information in the right-hand table.)
+          </p>
+
+<codeblock><![CDATA[SELECT c.c_custkey, c.c_name
+  FROM customer c LEFT ANTI JOIN c.c_orders o
+LIMIT 5;
++-----------+--------------------+
+| c_custkey | c_name             |
++-----------+--------------------+
+| 60210     | Customer#000060210 |
+| 147873    | Customer#000147873 |
+| 141576    | Customer#000141576 |
+| 85365     | Customer#000085365 |
+| 70998     | Customer#000070998 |
++-----------+--------------------+
+]]>
+</codeblock>
+
+<!-- To do: promote the correlated subquery aspect into its own subtopic. -->
+
+          <p>
+            You can also perform correlated subqueries to examine the properties of complex type columns for each row in the result set.
+          </p>
+
+          <p>
+            Count the number of orders per customer. Note the correlated reference to the table alias <codeph>C</codeph>. The
+            <codeph>COUNT(*)</codeph> operation applies to all the elements of the <codeph>C_ORDERS</codeph> array for the corresponding
+            row, avoiding the need for a <codeph>GROUP BY</codeph> clause.
+          </p>
+
+<codeblock>select c_name, howmany FROM customer c, (SELECT COUNT(*) howmany FROM c.c_orders) v limit 5;
++--------------------+---------+
+| c_name             | howmany |
++--------------------+---------+
+| Customer#000030065 | 15      |
+| Customer#000065455 | 18      |
+| Customer#000113644 | 21      |
+| Customer#000111078 | 0       |
+| Customer#000024621 | 0       |
++--------------------+---------+
+</codeblock>
+
+          <p>
+            Count the number of orders per customer, ignoring any customers that have not placed any orders:
+          </p>
+
+<codeblock>SELECT c_name, howmany_orders
+FROM
+  customer c,
+  (SELECT COUNT(*) howmany_orders FROM c.c_orders) subq1
+WHERE howmany_orders > 0
+LIMIT 5;
++--------------------+----------------+
+| c_name             | howmany_orders |
++--------------------+----------------+
+| Customer#000072578 | 7              |
+| Customer#000046378 | 26             |
+| Customer#000069815 | 11             |
+| Customer#000079058 | 12             |
+| Customer#000092239 | 26             |
++--------------------+----------------+
+</codeblock>
+
+          <p>
+            Count the number of line items in each order. The reference to <codeph>C.C_ORDERS</codeph> in the <codeph>FROM</codeph> clause
+            is needed because the <codeph>O_ORDERKEY</codeph> field is a member of the elements in the <codeph>C_ORDERS</codeph> array. The
+            subquery labelled <codeph>SUBQ1</codeph> is correlated: it is re-evaluated for the <codeph>C_ORDERS.O_LINEITEMS</codeph> array
+            from each row of the <codeph>CUSTOMERS</codeph> table.
+          </p>
+
+<codeblock>SELECT c_name, o_orderkey, howmany_line_items
+FROM
+  customer c,
+  c.c_orders t2,
+  (SELECT COUNT(*) howmany_line_items FROM c.c_orders.o_lineitems) subq1
+WHERE howmany_line_items > 0
+LIMIT 5;
++--------------------+------------+--------------------+
+| c_name             | o_orderkey | howmany_line_items |
++--------------------+------------+--------------------+
+| Customer#000020890 | 1884930    | 95                 |
+| Customer#000020890 | 4570754    | 95                 |
+| Customer#000020890 | 3771072    | 95                 |
+| Customer#000020890 | 2555489    | 95                 |
+| Customer#000020890 | 919171     | 95                 |
++--------------------+------------+--------------------+
+</codeblock>
+
+          <p>
+            Get the number of orders, the average order price, and the maximum items in any order per customer. For this example, the
+            subqueries labelled <codeph>SUBQ1</codeph> and <codeph>SUBQ2</codeph> are correlated: they are re-evaluated for each row from
+            the original <codeph>CUSTOMER</codeph> table, and only apply to the complex columns associated with that row.
+          </p>
+
+<codeblock>SELECT c_name, howmany, average_price, most_items
+FROM
+  customer c,
+  (SELECT COUNT(*) howmany, AVG(o_totalprice) average_price FROM c.c_orders) subq1,
+  (SELECT MAX(l_quantity) most_items FROM c.c_orders.o_lineitems ) subq2
+LIMIT 5;
++--------------------+---------+---------------+------------+
+| c_name             | howmany | average_price | most_items |
++--------------------+---------+---------------+------------+
+| Customer#000030065 | 15      | 128908.34     | 50.00      |
+| Customer#000088191 | 0       | NULL          | NULL       |
+| Customer#000101555 | 10      | 164250.31     | 50.00      |
+| Customer#000022092 | 0       | NULL          | NULL       |
+| Customer#000036277 | 27      | 166040.06     | 50.00      |
++--------------------+---------+---------------+------------+
+</codeblock>
+
+          <p>
+            For example, these queries show how to access information about the <codeph>ARRAY</codeph> elements within the
+            <codeph>CUSTOMER</codeph> table from the <q>nested TPC-H</q> schema, starting with the initial <codeph>ARRAY</codeph> elements
+            and progressing to examine the <codeph>STRUCT</codeph> fields of the <codeph>ARRAY</codeph>, and then the elements nested within
+            another <codeph>ARRAY</codeph> of <codeph>STRUCT</codeph>:
+          </p>
+
+<codeblock><![CDATA[-- How many orders does each customer have?
+-- The type of the ARRAY column doesn't matter, this is just counting the elements.
+SELECT c_custkey, count(*)
+  FROM customer, customer.c_orders
+GROUP BY c_custkey
+LIMIT 5;
++-----------+----------+
+| c_custkey | count(*) |
++-----------+----------+
+| 61081     | 21       |
+| 115987    | 15       |
+| 69685     | 19       |
+| 109124    | 15       |
+| 50491     | 12       |
++-----------+----------+
+
+-- How many line items are part of each customer order?
+-- Now we examine a field from a STRUCT nested inside the ARRAY.
+SELECT c_custkey, c_orders.o_orderkey, count(*)
+  FROM customer, customer.c_orders c_orders, c_orders.o_lineitems
+GROUP BY c_custkey, c_orders.o_orderkey
+LIMIT 5;
++-----------+------------+----------+
+| c_custkey | o_orderkey | count(*) |
++-----------+------------+----------+
+| 63367     | 4985959    | 7        |
+| 53989     | 1972230    | 2        |
+| 143513    | 5750498    | 5        |
+| 17849     | 4857989    | 1        |
+| 89881     | 1046437    | 1        |
++-----------+------------+----------+
+
+-- What are the line items in each customer order?
+-- One of the STRUCT fields inside the ARRAY is another
+-- ARRAY containing STRUCT elements. The join finds
+-- all the related items from both levels of ARRAY.
+SELECT c_custkey, o_orderkey, l_partkey
+  FROM customer, customer.c_orders, c_orders.o_lineitems
+LIMIT 5;
++-----------+------------+-----------+
+| c_custkey | o_orderkey | l_partkey |
++-----------+------------+-----------+
+| 113644    | 2738497    | 175846    |
+| 113644    | 2738497    | 27309     |
+| 113644    | 2738497    | 175873    |
+| 113644    | 2738497    | 88559     |
+| 113644    | 2738497    | 8032      |
++-----------+------------+-----------+
+]]>
+</codeblock>
+
+        </conbody>
+
+      </concept>
+
+    </concept>
+
+    <concept id="pseudocolumns">
+
+      <title>Pseudocolumns for ARRAY and MAP Types</title>
+
+      <conbody>
+
+        <p>
+          Each element in an <codeph>ARRAY</codeph> type has a position, indexed starting from zero, and a value. Each element in a
+          <codeph>MAP</codeph> type represents a key-value pair. Impala provides pseudocolumns that let you retrieve this metadata as part
+          of a query, or filter query results by including such things in a <codeph>WHERE</codeph> clause. You refer to the pseudocolumns as
+          part of qualified column names in queries:
+        </p>
+
+        <ul>
+          <li>
+            <codeph>ITEM</codeph>: The value of an array element. If the <codeph>ARRAY</codeph> contains <codeph>STRUCT</codeph> elements,
+            you can refer to either <codeph><varname>array_name</varname>.ITEM.<varname>field_name</varname></codeph> or use the shorthand
+            <codeph><varname>array_name</varname>.<varname>field_name</varname></codeph>.
+          </li>
+
+          <li>
+            <codeph>POS</codeph>: The position of an element within an array.
+          </li>
+
+          <li>
+            <codeph>KEY</codeph>: The value forming the first part of a key-value pair in a map. It is not necessarily unique.
+          </li>
+
+          <li>
+            <codeph>VALUE</codeph>: The data item forming the second part of a key-value pair in a map. If the <codeph>VALUE</codeph> part
+            of the <codeph>MAP</codeph> element is a <codeph>STRUCT</codeph>, you can refer to either
+            <codeph><varname>map_name</varname>.VALUE.<varname>field_name</varname></codeph> or use the shorthand
+            <codeph><varname>map_name</varname>.<varname>field_name</varname></codeph>.
+          </li>
+        </ul>
+
+<!-- To do: Consider whether to move the detailed subtopics underneath ARRAY and MAP instead of embedded here. -->
+
+        <p outputclass="toc inpage"/>
+
+      </conbody>
+
+      <concept id="item">
+
+        <title id="pos">ITEM and POS Pseudocolumns</title>
+
+        <conbody>
+
+          <p>
+            When an <codeph>ARRAY</codeph> column contains <codeph>STRUCT</codeph> elements, you can refer to a field within the
+            <codeph>STRUCT</codeph> using a qualified name of the form
+            <codeph><varname>array_column</varname>.<varname>field_name</varname></codeph>. If the <codeph>ARRAY</codeph> contains scalar
+            values, Impala recognizes the special name <codeph><varname>array_column</varname>.ITEM</codeph> to represent the value of each
+            scalar array element. For example, if a column contained an <codeph>ARRAY</codeph> where each element was a
+            <codeph>STRING</codeph>, you would use <codeph><varname>array_name</varname>.ITEM</codeph> to refer to each scalar value in the
+            <codeph>SELECT</codeph> list, or the <codeph>WHERE</codeph> or other clauses.
+          </p>
+
+          <p>
+            This example shows a table with two <codeph>ARRAY</codeph> columns whose elements are of the scalar type
+            <codeph>STRING</codeph>. When referring to the values of the array elements in the <codeph>SELECT</codeph> list,
+            <codeph>WHERE</codeph> clause, or <codeph>ORDER BY</codeph> clause, you use the <codeph>ITEM</codeph> pseudocolumn because
+            within the array, the individual elements have no defined names.
+          </p>
+
+<codeblock><![CDATA[create TABLE persons_of_interest
+(
+person_id BIGINT,
+aliases ARRAY <STRING>,
+associates ARRAY <STRING>,
+real_name STRING
+)
+STORED AS PARQUET;
+
+-- Get all the aliases of each person.
+SELECT real_name, aliases.ITEM
+  FROM persons_of_interest, persons_of_interest.aliases
+ORDER BY real_name, aliases.item;
+
+-- Search for particular associates of each person.
+SELECT real_name, associates.ITEM
+  FROM persons_of_interest, persons_of_interest.associates
+WHERE associates.item LIKE '% MacGuffin';
+]]>
+</codeblock>
+
+          <p>
+            Because an array is inherently an ordered data structure, Impala recognizes the special name
+            <codeph><varname>array_column</varname>.POS</codeph> to represent the numeric position of each element within the array. The
+            <codeph>POS</codeph> pseudocolumn lets you filter or reorder the result set based on the sequence of array elements.
+          </p>
+
+          <p>
+            The following example uses a table from a flattened version of the TPC-H schema. The <codeph>REGION</codeph> table only has a
+            few rows, such as one row for Europe and one for Asia. The row for each region represents all the countries in that region as an
+            <codeph>ARRAY</codeph> of <codeph>STRUCT</codeph> elements:
+          </p>
+
+<codeblock><![CDATA[[localhost:21000] > desc region;
++-------------+--------------------------------------------------------------------+
+| name        | type                                                               |
++-------------+--------------------------------------------------------------------+
+| r_regionkey | smallint                                                           |
+| r_name      | string                                                             |
+| r_comment   | string                                                             |
+| r_nations   | array<struct<n_nationkey:smallint,n_name:string,n_comment:string>> |
++-------------+--------------------------------------------------------------------+
+]]>
+</codeblock>
+
+          <p>
+            To find the countries within a specific region, you use a join query. To find out the order of elements in the array, you also
+            refer to the <codeph>POS</codeph> pseudocolumn in the select list:
+          </p>
+
+<codeblock>[localhost:21000] > SELECT r1.r_name, r2.n_name, <b>r2.POS</b>
+                  > FROM region r1 INNER JOIN r1.r_nations r2
+                  > WHERE r1.r_name = 'ASIA';
++--------+-----------+-----+
+| r_name | n_name    | pos |
++--------+-----------+-----+
+| ASIA   | VIETNAM   | 0   |
+| ASIA   | CHINA     | 1   |
+| ASIA   | JAPAN     | 2   |
+| ASIA   | INDONESIA | 3   |
+| ASIA   | INDIA     | 4   |
++--------+-----------+-----+
+</codeblock>
+
+          <p>
+            Once you know the positions of the elements, you can use that information in subsequent queries, for example to change the
+            ordering of results from the complex type column or to filter certain elements from the array:
+          </p>
+
+<codeblock>[localhost:21000] > SELECT r1.r_name, r2.n_name, r2.POS
+                  > FROM region r1 INNER JOIN r1.r_nations r2
+                  > WHERE r1.r_name = 'ASIA'
+                  > <b>ORDER BY r2.POS DESC</b>;
++--------+-----------+-----+
+| r_name | n_name    | pos |
++--------+-----------+-----+
+| ASIA   | INDIA     | 4   |
+| ASIA   | INDONESIA | 3   |
+| ASIA   | JAPAN     | 2   |
+| ASIA   | CHINA     | 1   |
+| ASIA   | VIETNAM   | 0   |
++--------+-----------+-----+
+[localhost:21000] > SELECT r1.r_name, r2.n_name, r2.POS
+                  > FROM region r1 INNER JOIN r1.r_nations r2
+                  > WHERE r1.r_name = 'ASIA' AND <b>r2.POS BETWEEN 1 and 3</b>;
++--------+-----------+-----+
+| r_name | n_name    | pos |
++--------+-----------+-----+
+| ASIA   | CHINA     | 1   |
+| ASIA   | JAPAN     | 2   |
+| ASIA   | INDONESIA | 3   |
++--------+-----------+-----+
+</codeblock>
+
+        </conbody>
+
+      </concept>
+
+      <concept id="key">
+
+        <title id="value">KEY and VALUE Pseudocolumns</title>
+
+        <conbody>
+
+          <p>
+            The <codeph>MAP</codeph> data type is suitable for representing sparse or wide data structures, where each row might only have
+            entries for a small subset of named fields. Because the element names (the map keys) vary depending on the row, a query must be
+            able to refer to both the key and the value parts of each key-value pair. The <codeph>KEY</codeph> and <codeph>VALUE</codeph>
+            pseudocolumns let you refer to the parts of the key-value pair independently within the query, as
+            <codeph><varname>map_column</varname>.KEY</codeph> and <codeph><varname>map_column</varname>.VALUE</codeph>.
+          </p>
+
+          <p>
+            The <codeph>KEY</codeph> must always be a scalar type, such as <codeph>STRING</codeph>, <codeph>BIGINT</codeph>, or
+            <codeph>TIMESTAMP</codeph>. It can be <codeph>NULL</codeph>. Values of the <codeph>KEY</codeph> field are not necessarily unique
+            within the same <codeph>MAP</codeph>. You apply any required <codeph>DISTINCT</codeph>, <codeph>GROUP BY</codeph>, and other
+            clauses in the query, and loop through the result set to process all the values matching any specified keys.
+          </p>
+
+          <p>
+            The <codeph>VALUE</codeph> can be either a scalar type or another complex type. If the <codeph>VALUE</codeph> is a
+            <codeph>STRUCT</codeph>, you can construct a qualified name
+            <codeph><varname>map_column</varname>.VALUE.<varname>struct_field</varname></codeph> to refer to the individual fields inside
+            the value part. If the <codeph>VALUE</codeph> is an <codeph>ARRAY</codeph> or another <codeph>MAP</codeph>, you must include
+            another join condition that establishes a table alias for <codeph><varname>map_column</varname>.VALUE</codeph>, and then
+            construct another qualified name using that alias, for example <codeph><varname>table_alias</varname>.ITEM</codeph> or
+            <codeph><varname>table_alias</varname>.KEY</codeph> and <codeph><varname>table_alias</varname>.VALUE</codeph>
+          </p>
+
+          <p>
+            The following example shows different ways to access a <codeph>MAP</codeph> column using the <codeph>KEY</codeph> and
+            <codeph>VALUE</codeph> pseudocolumns. The <codeph>DETAILS</codeph> column has a <codeph>STRING</codeph> first part with short,
+            standardized values such as <codeph>'Recurring'</codeph>, <codeph>'Lucid'</codeph>, or <codeph>'Anxiety'</codeph>. This is the
+            <q>key</q> that is used to look up particular kinds of elements from the <codeph>MAP</codeph>. The second part, also a
+            <codeph>STRING</codeph>, is a longer free-form explanation. Impala gives you the standard pseudocolumn names
+            <codeph>KEY</codeph> and <codeph>VALUE</codeph> for the two parts, and you apply your own conventions and interpretations to the
+            underlying values.
+          </p>
+
+          <note>
+            If you find that the single-item nature of the <codeph>VALUE</codeph> makes it difficult to model your data accurately, the
+            solution is typically to add some nesting to the complex type. For example, to have several sets of key-value pairs, make the
+            column an <codeph>ARRAY</codeph> whose elements are <codeph>MAP</codeph>. To make a set of key-value pairs that holds more
+            elaborate information, make a <codeph>MAP</codeph> column whose <codeph>VALUE</codeph> part contains an <codeph>ARRAY</codeph>
+            or a <codeph>STRUCT</codeph>.
+          </note>
+
+<codeblock><![CDATA[CREATE TABLE dream_journal
+(
+  dream_id BIGINT,
+  details MAP <STRING,STRING>
+)
+STORED AS PARQUET;
+]]>
+
+-- What are all the types of dreams that are recorded?
+SELECT DISTINCT details.KEY FROM dream_journal, dream_journal.details;
+
+-- How many lucid dreams were recorded?
+-- Because there is no GROUP BY, we count the 'Lucid' keys across all rows.
+SELECT <b>COUNT(details.KEY)</b>
+  FROM dream_journal, dream_journal.details
+WHERE <b>details.KEY = 'Lucid'</b>;
+
+-- Print a report of a subset of dreams, filtering based on both the lookup key
+-- and the detailed value.
+SELECT dream_id, <b>details.KEY AS "Dream Type"</b>, <b>details.VALUE AS "Dream Summary"</b>
+  FROM dream_journal, dream_journal.details
+WHERE
+  <b>details.KEY IN ('Happy', 'Pleasant', 'Joyous')</b>
+  AND <b>details.VALUE LIKE '%childhood%'</b>;
+</codeblock>
+
+          <p>
+            The following example shows a more elaborate version of the previous table, where the <codeph>VALUE</codeph> part of the
+            <codeph>MAP</codeph> entry is a <codeph>STRUCT</codeph> rather than a scalar type. Now instead of referring to the
+            <codeph>VALUE</codeph> pseudocolumn directly, you use dot notation to refer to the <codeph>STRUCT</codeph> fields inside it.
+          </p>
+
+<codeblock><![CDATA[CREATE TABLE better_dream_journal
+(
+  dream_id BIGINT,
+  details MAP <STRING,STRUCT <summary: STRING, when_happened: TIMESTAMP, duration: DECIMAL(5,2), woke_up: BOOLEAN> >
+)
+STORED AS PARQUET;
+]]>
+
+-- Do more elaborate reporting and filtering by examining multiple attributes within the same dream.
+SELECT dream_id, <b>details.KEY AS "Dream Type"</b>, <b>details.VALUE.summary AS "Dream Summary"</b>, <b>details.VALUE.duration AS "Duration"</b>
+  FROM better_dream_journal, better_dream_journal.details
+WHERE
+  <b>details.KEY IN ('Anxiety', 'Nightmare')</b>
+  AND <b>details.VALUE.duration > 60</b>
+  AND <b>details.VALUE.woke_up = TRUE</b>;
+
+-- Remember that if the ITEM or VALUE contains a STRUCT, you can reference
+-- the STRUCT fields directly without the .ITEM or .VALUE qualifier.
+SELECT dream_id, <b>details.KEY AS "Dream Type"</b>, <b>details.summary AS "Dream Summary"</b>, <b>details.duration AS "Duration"</b>
+  FROM better_dream_journal, better_dream_journal.details
+WHERE
+  <b>details.KEY IN ('Anxiety', 'Nightmare')</b>
+  AND <b>details.duration > 60</b>
+  AND <b>details.woke_up = TRUE</b>;
+</codeblock>
+
+        </conbody>
+
+      </concept>
+
+    </concept>
+
+    <concept id="complex_types_etl">
+
+<!-- This topic overlaps in many ways with the preceding one. See which theme resonates with users, and combine them under the better title. -->
+
+      <title>Loading Data Containing Complex Types</title>
+
+      <conbody>
+
+        <p>
+          Because the Impala <codeph>INSERT</codeph> statement does not currently support creating new data with complex type columns, or
+          copying existing complex type values from one table to another, you primarily use Impala to query Parquet tables with complex
+          types where the data was inserted through Hive, or create tables with complex types where you already have existing Parquet data
+          files.
+        </p>
+
+        <p>
+          If you have created a Hive table with the Parquet file format and containing complex types, use the same table for Impala queries
+          with no changes. If you have such a Hive table in some other format, use a Hive <codeph>CREATE TABLE AS SELECT ... STORED AS
+          PARQUET</codeph> or <codeph>INSERT ... SELECT</codeph> statement to produce an equivalent Parquet table that Impala can query.
+        </p>
+
+        <p>
+          If you have existing Parquet data files containing complex types, located outside of any Impala or Hive table, such as data files
+          created by Spark jobs, you can use an Impala <codeph>CREATE TABLE ... STORED AS PARQUET</codeph> statement, followed by an Impala
+          <codeph>LOAD DATA</codeph> statement to move the data files into the table. As an alternative, you can use an Impala
+          <codeph>CREATE EXTERNAL TABLE</codeph> statement to create a table pointing to the HDFS directory that already contains the data
+          files.
+        </p>
+
+        <p>
+          Perhaps the simplest way to get started with complex type data is to take a denormalized table containing duplicated values, and
+          use an <codeph>INSERT ... SELECT</codeph> statement to copy the data into a Parquet table and condense the repeated values into
+          complex types. With the Hive <codeph>INSERT</codeph> statement, you use the <codeph>COLLECT_LIST()</codeph>,
+          <codeph>NAMED_STRUCT()</codeph>, and <codeph>MAP()</codeph> constructor functions within a <codeph>GROUP BY</codeph> query to
+          produce the complex type values. <codeph>COLLECT_LIST()</codeph> turns a sequence of values into an <codeph>ARRAY</codeph>.
+          <codeph>NAMED_STRUCT()</codeph> uses the first, third, and so on arguments as the field names for a <codeph>STRUCT</codeph>, to
+          match the field names from the <codeph>CREATE TABLE</codeph> statement.
+        </p>
+
+        <note>
+          Because Hive currently cannot construct individual rows using complex types through the <codeph>INSERT ... VALUES</codeph> syntax,
+          you prepare the data in flat form in a separate table, then copy it to the table with complex columns using <codeph>INSERT ...
+          SELECT</codeph> and the complex type constructors. See <xref href="impala_complex_types.xml#complex_types_ex_hive_etl"/> for
+          examples.
+        </note>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="complex_types_nesting">
+
+      <title>Using Complex Types as Nested Types</title>
+
+      <conbody>
+
+        <p>
+          The <codeph>ARRAY</codeph>, <codeph>STRUCT</codeph>, and <codeph>MAP</codeph> types can be the top-level types for <q>nested
+          type</q> columns. That is, each of these types can contain other complex or scalar types, with multiple levels of nesting to a
+          maximum depth of 100. For example, you can have an array of structures, a map containing other maps, a structure containing an
+          array of other structures, and so on. At the lowest level, there are always scalar types making up the fields of a
+          <codeph>STRUCT</codeph>, elements of an <codeph>ARRAY</codeph>, and keys and values of a <codeph>MAP</codeph>.
+        </p>
+
+        <p>
+          Schemas involving complex types typically use some level of nesting for the complex type columns.
+        </p>
+
+        <p>
+          For example, to model a relationship like a dimension table and a fact table, you typically use an <codeph>ARRAY</codeph> where
+          each array element is a <codeph>STRUCT</codeph>. The <codeph>STRUCT</codeph> fields represent what would traditionally be columns
+          in a separate joined table. It makes little sense to use a <codeph>STRUCT</codeph> as the top-level type for a column, because you
+          could just make the fields of the <codeph>STRUCT</codeph> into regular table columns.
+        </p>
+
+<!-- To do: this example might move somewhere else, under STRUCT itself or in a tips-and-tricks section. -->
+
+        <p>
+          Perhaps the only use case for a top-level <codeph>STRUCT</codeph> would be to to allow <codeph>STRUCT</codeph> fields with the
+          same name as columns to coexist in the same table. The following example shows how a table could have a column named
+          <codeph>ID</codeph>, and two separate <codeph>STRUCT</codeph> fields also named <codeph>ID</codeph>. Because the
+          <codeph>STRUCT</codeph> fields are always referenced using qualified names, the identical <codeph>ID</codeph> names do not cause a
+          conflict.
+        </p>
+
+<codeblock><![CDATA[CREATE TABLE struct_namespaces
+(
+  id BIGINT
+  , s1 STRUCT < id: BIGINT, field1: STRING >
+  , s2 STRUCT < id: BIGINT, when_happened: TIMESTAMP >
+)
+STORED AS PARQUET;
+
+select id, s1.id, s2.id from struct_namespaces;
+]]>
+</codeblock>
+
+        <p>
+          It is common to make the value portion of each key-value pair in a <codeph>MAP</codeph> a <codeph>STRUCT</codeph>,
+          <codeph>ARRAY</codeph> of <codeph>STRUCT</codeph>, or other complex type variation. That way, each key in the <codeph>MAP</codeph>
+          can be associated with a flexible and extensible data structure. The key values are not predefined ahead of time (other than by
+          specifying their data type). Therefore, the <codeph>MAP</codeph> can accomodate a rapidly evolving schema, or sparse data
+          structures where each row contains only a few data values drawn from a large set of possible choices.
+        </p>
+
+        <p>
+          Although you can use an <codeph>ARRAY</codeph> of scalar values as the top-level column in a table, such a simple array is
+          typically of limited use for analytic queries. The only property of the array elements, aside from the element value, is the
+          ordering sequence available through the <codeph>POS</codeph> pseudocolumn. To record any additional item about each array element,
+          such as a <codeph>TIMESTAMP</codeph> or a symbolic name, you use an <codeph>ARRAY</codeph> of <codeph>STRUCT</codeph> rather than
+          of scalar values.
+        </p>
+
+        <p>
+          If you are considering having multiple <codeph>ARRAY</codeph> or <codeph>MAP</codeph> columns, with related items under the same
+          position in each <codeph>ARRAY</codeph> or the same key in each <codeph>MAP</codeph>, prefer to use a <codeph>STRUCT</codeph> to
+          group all the related items into a single <codeph>ARRAY</codeph> or <codeph>MAP</codeph>. Doing so avoids the additional storage
+          overhead and potential duplication of key values from having an extra complex type column. Also, because each
+          <codeph>ARRAY</codeph> or <codeph>MAP</codeph> that you reference in the query <codeph>SELECT</codeph> list requires an additional
+          join clause, minimizing the number of complex type columns also makes the query easier to read and maintain, relying more on dot
+          notation to refer to the relevant fields rather than a sequence of join clauses.
+        </p>
+
+        <p>
+          For example, here is a table with several complex type columns all at the top level and containing only scalar types. To retrieve
+          every data item for the row requires a separate join for each <codeph>ARRAY</codeph> or <codeph>MAP</codeph> column. The fields of
+          the <codeph>STRUCT</codeph> can be referenced using dot notation, but there is no real advantage to using the
+          <codeph>STRUCT</codeph> at the top level rather than just making separate columns <codeph>FIELD1</codeph> and
+          <codeph>FIELD2</codeph>.
+        </p>
+
+<codeblock><![CDATA[CREATE TABLE complex_types_top_level
+(
+  id BIGINT,
+  a1 ARRAY<INT>,
+  a2 ARRAY<STRING>,
+  s STRUCT<field1: INT, field2: STRING>,
+-- Numeric lookup key for a string value.
+  m1 MAP<INT,STRING>,
+-- String lookup key for a numeric value.
+  m2 MAP<STRING,INT>
+)
+STORED AS PARQUET;
+
+describe complex_types_top_level;
++------+-----------------+
+| name | type            |
++------+-----------------+
+| id   | bigint          |
+| a1   | array<int>      |
+| a2   | array<string>   |
+| s    | struct<         |
+|      |   field1:int,   |
+|      |   field2:string |
+|      | >               |
+| m1   | map<int,string> |
+| m2   | map<string,int> |
++------+-----------------+
+
+select
+  id,
+  a1.item,
+  a2.item,
+  s.field1,
+  s.field2,
+  m1.key,
+  m1.value,
+  m2.key,
+  m2.value
+from
+  complex_types_top_level,
+  complex_types_top_level.a1,
+  complex_types_top_level.a2,
+  complex_types_top_level.m1,
+  complex_types_top_level.m2;
+]]>
+</codeblock>
+
+        <p>
+          For example, here is a table with columns containing an <codeph>ARRAY</codeph> of <codeph>STRUCT</codeph>, a <codeph>MAP</codeph>
+          where each key value is a <codeph>STRUCT</codeph>, and a <codeph>MAP</codeph> where each key value is an <codeph>ARRAY</codeph> of
+          <codeph>STRUCT</codeph>.
+        </p>
+
+<codeblock><![CDATA[CREATE TABLE nesting_demo
+(
+  user_id BIGINT,
+  family_members ARRAY < STRUCT < name: STRING, email: STRING, date_joined: TIMESTAMP >>,
+  foo map < STRING, STRUCT < f1: INT, f2: INT, f3: TIMESTAMP, f4: BOOLEAN >>,
+  gameplay MAP < STRING , ARRAY < STRUCT <
+    name: STRING, highest: BIGINT, lives_used: INT, total_spent: DECIMAL(16,2)
+  >>>
+)
+STORED AS PARQUET;
+]]>
+</codeblock>
+
+        <p>
+          The <codeph>DESCRIBE</codeph> statement rearranges the <codeph>&lt;</codeph> and <codeph>&gt;</codeph> separato

<TRUNCATED>


[13/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_perf_stats.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_perf_stats.xml b/docs/topics/impala_perf_stats.xml
new file mode 100644
index 0000000..33d394d
--- /dev/null
+++ b/docs/topics/impala_perf_stats.xml
@@ -0,0 +1,1031 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="perf_stats">
+
+  <title>Table and Column Statistics</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Concepts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Impala can do better optimization for complex or multi-table queries when it has access to statistics about
+      the volume of data and how the values are distributed. Impala uses this information to help parallelize and
+      distribute the work for a query. For example, optimizing join queries requires a way of determining if one
+      table is <q>bigger</q> than another, which is a function of the number of rows and the average row size
+      for each table. The following sections describe the categories of statistics Impala can work
+      with, and how to produce them and keep them up to date.
+    </p>
+
+    <note>
+      <p rev="1.2.2">
+        Originally, Impala relied on the Hive mechanism for collecting statistics, through the Hive <codeph>ANALYZE
+        TABLE</codeph> statement which initiates a MapReduce job. For better user-friendliness and reliability,
+        Impala implements its own <codeph>COMPUTE STATS</codeph> statement in Impala 1.2.2 and higher, along with the
+        <codeph>DROP STATS</codeph>, <codeph>SHOW TABLE STATS</codeph>, and <codeph>SHOW COLUMN STATS</codeph>
+        statements.
+      </p>
+    </note>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="perf_table_stats">
+
+    <title id="table_stats">Overview of Table Statistics</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Concepts"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+<!-- Hive background info: https://cwiki.apache.org/Hive/statsdev.html -->
+
+      <p>
+        The Impala query planner can make use of statistics about entire tables and partitions.
+        This information includes physical characteristics such as the number of rows, number of data files,
+        the total size of the data files, and the file format. For partitioned tables, the numbers
+        are calculated per partition, and as totals for the whole table.
+        This metadata is stored in the metastore database, and can be updated by either Impala or Hive.
+        If a number is not available, the value -1 is used as a placeholder.
+        Some numbers, such as number and total sizes of data files, are always kept up to date because
+        they can be calculated cheaply, as part of gathering HDFS block metadata.
+      </p>
+
+      <p>
+        The following example shows table stats for an unpartitioned Parquet table.
+        The values for the number and sizes of files are always available.
+        Initially, the number of rows is not known, because it requires a potentially expensive
+        scan through the entire table, and so that value is displayed as -1.
+        The <codeph>COMPUTE STATS</codeph> statement fills in any unknown table stats values.
+      </p>
+
+<codeblock>
+show table stats parquet_snappy;
++-------+--------+---------+--------------+-------------------+---------+-------------------+...
+| #Rows | #Files | Size    | Bytes Cached | Cache Replication | Format  | Incremental stats |...
++-------+--------+---------+--------------+-------------------+---------+-------------------+...
+| -1    | 96     | 23.35GB | NOT CACHED   | NOT CACHED        | PARQUET | false             |...
++-------+--------+---------+--------------+-------------------+---------+-------------------+...
+
+compute stats parquet_snappy;
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 1 partition(s) and 6 column(s). |
++-----------------------------------------+
+
+
+show table stats parquet_snappy;
++------------+--------+---------+--------------+-------------------+---------+-------------------+...
+| #Rows      | #Files | Size    | Bytes Cached | Cache Replication | Format  | Incremental stats |...
++------------+--------+---------+--------------+-------------------+---------+-------------------+...
+| 1000000000 | 96     | 23.35GB | NOT CACHED   | NOT CACHED        | PARQUET | false             |...
++------------+--------+---------+--------------+-------------------+---------+-------------------+...
+</codeblock>
+
+      <p>
+        Impala performs some optimizations using this metadata on its own, and other optimizations by
+        using a combination of table and column statistics.
+      </p>
+
+      <p rev="1.2.1">
+        To check that table statistics are available for a table, and see the details of those statistics, use the
+        statement <codeph>SHOW TABLE STATS <varname>table_name</varname></codeph>. See
+        <xref href="impala_show.xml#show"/> for details.
+      </p>
+
+      <p>
+        If you use the Hive-based methods of gathering statistics, see
+        <xref href="https://cwiki.apache.org/confluence/display/Hive/StatsDev" scope="external" format="html">the
+        Hive wiki</xref> for information about the required configuration on the Hive side. <ph rev="upstream">Cloudera</ph> recommends
+        using the Impala <codeph>COMPUTE STATS</codeph> statement to avoid potential configuration and scalability
+        issues with the statistics-gathering process.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/hive_column_stats_caveat"/>
+    </conbody>
+  </concept>
+
+  <concept id="perf_column_stats">
+
+    <title id="column_stats">Overview of Column Statistics</title>
+
+    <conbody>
+
+<!-- Cloudera+Hive background information: http://blog.cloudera.com/blog/2012/08/column-statistics-in-hive/ -->
+
+      <p>
+        The Impala query planner can make use of statistics about individual columns when that metadata is
+        available in the metastore database. This technique is most valuable for columns compared across tables in
+        <xref href="impala_perf_joins.xml#perf_joins">join queries</xref>, to help estimate how many rows the query
+        will retrieve from each table. <ph rev="2.0.0"> These statistics are also important for correlated
+        subqueries using the <codeph>EXISTS()</codeph> or <codeph>IN()</codeph> operators, which are processed
+        internally the same way as join queries.</ph>
+      </p>
+
+      <p>
+        The following example shows column stats for an unpartitioned Parquet table.
+        The values for the maximum and average sizes of some types are always available,
+        because those figures are constant for numeric and other fixed-size types.
+        Initially, the number of distinct values is not known, because it requires a potentially expensive
+        scan through the entire table, and so that value is displayed as -1.
+        The same applies to maximum and average sizes of variable-sized types, such as <codeph>STRING</codeph>.
+        The <codeph>COMPUTE STATS</codeph> statement fills in most unknown column stats values.
+        (It does not record the number of <codeph>NULL</codeph> values, because currently Impala
+        does not use that figure for query optimization.)
+      </p>
+
+<codeblock>
+show column stats parquet_snappy;
++-------------+----------+------------------+--------+----------+----------+
+| Column      | Type     | #Distinct Values | #Nulls | Max Size | Avg Size |
++-------------+----------+------------------+--------+----------+----------+
+| id          | BIGINT   | -1               | -1     | 8        | 8        |
+| val         | INT      | -1               | -1     | 4        | 4        |
+| zerofill    | STRING   | -1               | -1     | -1       | -1       |
+| name        | STRING   | -1               | -1     | -1       | -1       |
+| assertion   | BOOLEAN  | -1               | -1     | 1        | 1        |
+| location_id | SMALLINT | -1               | -1     | 2        | 2        |
++-------------+----------+------------------+--------+----------+----------+
+
+compute stats parquet_snappy;
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 1 partition(s) and 6 column(s). |
++-----------------------------------------+
+
+show column stats parquet_snappy;
++-------------+----------+------------------+--------+----------+-------------------+
+| Column      | Type     | #Distinct Values | #Nulls | Max Size | Avg Size          |
++-------------+----------+------------------+--------+----------+-------------------+
+| id          | BIGINT   | 183861280        | -1     | 8        | 8                 |
+| val         | INT      | 139017           | -1     | 4        | 4                 |
+| zerofill    | STRING   | 101761           | -1     | 6        | 6                 |
+| name        | STRING   | 145636240        | -1     | 22       | 13.00020027160645 |
+| assertion   | BOOLEAN  | 2                | -1     | 1        | 1                 |
+| location_id | SMALLINT | 339              | -1     | 2        | 2                 |
++-------------+----------+------------------+--------+----------+-------------------+
+</codeblock>
+
+      <note>
+        <p>
+          For column statistics to be effective in Impala, you also need to have table statistics for the
+          applicable tables, as described in <xref href="impala_perf_stats.xml#perf_table_stats"/>. When you use
+          the Impala <codeph>COMPUTE STATS</codeph> statement, both table and column statistics are automatically
+          gathered at the same time, for all columns in the table.
+        </p>
+        <p conref="../shared/impala_common.xml#common/decimal_no_stats"/>
+      </note>
+
+      <note conref="../shared/impala_common.xml#common/compute_stats_nulls"/>
+
+<!-- Hive-based instructions are considered obsolete since the introduction of the Impala COMPUTE STATS statement.
+      <p>
+        Add settings like the following to the <filepath>hive-site.xml</filepath>
+        configuration file, in the Hive configuration directory, on every node where you run
+        <codeph>ANALYZE TABLE</codeph> statements through the
+        <codeph>hive</codeph> shell. The
+        <codeph>hive.stats.ndv.error</codeph> setting represents the standard error when
+        estimating the number of distinct values for a column. The value of 5.0 is recommended as a tradeoff between the
+        accuracy of the gathered statistics and the resource usage of the stats-gathering process.
+      </p>
+
+<codeblock><![CDATA[<property>
+ <name>hive.stats.ndv.error</name>
+ <value>5.0</value>
+</property>]]></codeblock>
+
+      <p>
+        5.0 is a relatively low value that devotes substantial computational resources to the statistics-gathering
+        process. To reduce the resource usage, you could increase this value; to make the statistics even more precise,
+        you could lower it.
+      </p>
+
+      <p>
+        The syntax for gathering column statistics uses the <codeph>ANALYZE TABLE ...
+        COMPUTE STATISTICS</codeph> clause, with an additional <codeph>FOR
+        COLUMNS</codeph> clause. For partitioned tables, you can gather statistics for specific partitions by including
+        a clause <codeph>PARTITION
+        (<varname>col1=val1</varname>,<varname>col2=val2</varname>,
+        ...)</codeph>; but you cannot include the partitioning columns in the
+        <codeph>FOR COLUMNS</codeph> clause. Also, you cannot use fully qualified table
+        names, so issue a <codeph>USE</codeph> command first to switch to the
+        appropriate database. For example:
+      </p>
+
+<codeblock>USE <varname>database_name</varname>;
+ANALYZE TABLE <varname>table_name</varname> COMPUTE STATISTICS FOR COLUMNS <varname>column_list</varname>;
+ANALYZE TABLE <varname>table_name</varname> PARTITION (<varname>partition_specs</varname>) COMPUTE STATISTICS FOR COLUMNS <varname>column_list</varname>;</codeblock>
+-->
+
+      <p rev="1.2.1">
+        To check whether column statistics are available for a particular set of columns, use the <codeph>SHOW
+        COLUMN STATS <varname>table_name</varname></codeph> statement, or check the extended
+        <codeph>EXPLAIN</codeph> output for a query against that table that refers to those columns. See
+        <xref href="impala_show.xml#show"/> and <xref href="impala_explain.xml#explain"/> for details.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/hive_column_stats_caveat"/>
+    </conbody>
+  </concept>
+
+  <concept id="perf_stats_partitions">
+    <title id="stats_partitions">How Table and Column Statistics Work for Partitioned Tables</title>
+    <conbody>
+
+      <p>
+        When you use Impala for <q>big data</q>, you are highly likely to use partitioning
+        for your biggest tables, the ones representing data that can be logically divided
+        based on dates, geographic regions, or similar criteria. The table and column statistics
+        are especially useful for optimizing queries on such tables. For example, a query involving
+        one year might involve substantially more or less data than a query involving a different year,
+        or a range of several years. Each query might be optimized differently as a result.
+      </p>
+
+      <p>
+        The following examples show how table and column stats work with a partitioned table.
+        The table for this example is partitioned by year, month, and day.
+        For simplicity, the sample data consists of 5 partitions, all from the same year and month.
+        Table stats are collected independently for each partition. (In fact, the
+        <codeph>SHOW PARTITIONS</codeph> statement displays exactly the same information as
+        <codeph>SHOW TABLE STATS</codeph> for a partitioned table.) Column stats apply to
+        the entire table, not to individual partitions. Because the partition key column values
+        are represented as HDFS directories, their characteristics are typically known in advance,
+        even when the values for non-key columns are shown as -1.
+      </p>
+
+<codeblock>
+show partitions year_month_day;
++-------+-------+-----+-------+--------+---------+--------------+-------------------+---------+...
+| year  | month | day | #Rows | #Files | Size    | Bytes Cached | Cache Replication | Format  |...
++-------+-------+-----+-------+--------+---------+--------------+-------------------+---------+...
+| 2013  | 12    | 1   | -1    | 1      | 2.51MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 2   | -1    | 1      | 2.53MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 3   | -1    | 1      | 2.52MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 4   | -1    | 1      | 2.51MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 5   | -1    | 1      | 2.52MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| Total |       |     | -1    | 5      | 12.58MB | 0B           |                   |         |...
++-------+-------+-----+-------+--------+---------+--------------+-------------------+---------+...
+
+show table stats year_month_day;
++-------+-------+-----+-------+--------+---------+--------------+-------------------+---------+...
+| year  | month | day | #Rows | #Files | Size    | Bytes Cached | Cache Replication | Format  |...
++-------+-------+-----+-------+--------+---------+--------------+-------------------+---------+...
+| 2013  | 12    | 1   | -1    | 1      | 2.51MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 2   | -1    | 1      | 2.53MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 3   | -1    | 1      | 2.52MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 4   | -1    | 1      | 2.51MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 5   | -1    | 1      | 2.52MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| Total |       |     | -1    | 5      | 12.58MB | 0B           |                   |         |...
++-------+-------+-----+-------+--------+---------+--------------+-------------------+---------+...
+
+show column stats year_month_day;
++-----------+---------+------------------+--------+----------+----------+
+| Column    | Type    | #Distinct Values | #Nulls | Max Size | Avg Size |
++-----------+---------+------------------+--------+----------+----------+
+| id        | INT     | -1               | -1     | 4        | 4        |
+| val       | INT     | -1               | -1     | 4        | 4        |
+| zfill     | STRING  | -1               | -1     | -1       | -1       |
+| name      | STRING  | -1               | -1     | -1       | -1       |
+| assertion | BOOLEAN | -1               | -1     | 1        | 1        |
+| year      | INT     | 1                | 0      | 4        | 4        |
+| month     | INT     | 1                | 0      | 4        | 4        |
+| day       | INT     | 5                | 0      | 4        | 4        |
++-----------+---------+------------------+--------+----------+----------+
+
+compute stats year_month_day;
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 5 partition(s) and 5 column(s). |
++-----------------------------------------+
+
+show table stats year_month_day;
++-------+-------+-----+--------+--------+---------+--------------+-------------------+---------+...
+| year  | month | day | #Rows  | #Files | Size    | Bytes Cached | Cache Replication | Format  |...
++-------+-------+-----+--------+--------+---------+--------------+-------------------+---------+...
+| 2013  | 12    | 1   | 93606  | 1      | 2.51MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 2   | 94158  | 1      | 2.53MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 3   | 94122  | 1      | 2.52MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 4   | 93559  | 1      | 2.51MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| 2013  | 12    | 5   | 93845  | 1      | 2.52MB  | NOT CACHED   | NOT CACHED        | PARQUET |...
+| Total |       |     | 469290 | 5      | 12.58MB | 0B           |                   |         |...
++-------+-------+-----+--------+--------+---------+--------------+-------------------+---------+...
+
+show column stats year_month_day;
++-----------+---------+------------------+--------+----------+-------------------+
+| Column    | Type    | #Distinct Values | #Nulls | Max Size | Avg Size          |
++-----------+---------+------------------+--------+----------+-------------------+
+| id        | INT     | 511129           | -1     | 4        | 4                 |
+| val       | INT     | 364853           | -1     | 4        | 4                 |
+| zfill     | STRING  | 311430           | -1     | 6        | 6                 |
+| name      | STRING  | 471975           | -1     | 22       | 13.00160026550293 |
+| assertion | BOOLEAN | 2                | -1     | 1        | 1                 |
+| year      | INT     | 1                | 0      | 4        | 4                 |
+| month     | INT     | 1                | 0      | 4        | 4                 |
+| day       | INT     | 5                | 0      | 4        | 4                 |
++-----------+---------+------------------+--------+----------+-------------------+
+</codeblock>
+
+      <note>
+        Partitioned tables can grow so large that scanning the entire table, as the <codeph>COMPUTE STATS</codeph>
+        statement does, is impractical just to update the statistics for a new partition. The standard
+        <codeph>COMPUTE STATS</codeph> statement might take hours, or even days. That situation is where you switch
+        to using incremental statistics, a feature available in <keyword keyref="impala21_full"/> and higher.
+        See <xref href="impala_perf_stats.xml#perf_stats_incremental"/> for details about this feature
+        and the <codeph>COMPUTE INCREMENTAL STATS</codeph> syntax.
+      </note>
+
+      <p conref="../shared/impala_common.xml#common/hive_column_stats_caveat"/>
+    </conbody>
+  </concept>
+
+  <concept rev="2.1.0" id="perf_stats_incremental">
+
+    <title id="incremental_stats">Overview of Incremental Statistics</title>
+
+    <conbody>
+
+      <p>
+        In Impala 2.1.0 and higher, you can use the syntax <codeph>COMPUTE INCREMENTAL STATS</codeph> and
+        <codeph>DROP INCREMENTAL STATS</codeph>. The <codeph>INCREMENTAL</codeph> clauses work with incremental
+        statistics, a specialized feature for partitioned tables that are large or frequently updated with new
+        partitions.
+      </p>
+
+      <p>
+        When you compute incremental statistics for a partitioned table, by default Impala only processes those
+        partitions that do not yet have incremental statistics. By processing only newly added partitions, you can
+        keep statistics up to date for large partitioned tables, without incurring the overhead of reprocessing the
+        entire table each time.
+      </p>
+
+      <p>
+        You can also compute or drop statistics for a single partition by including a <codeph>PARTITION</codeph>
+        clause in the <codeph>COMPUTE INCREMENTAL STATS</codeph> or <codeph>DROP INCREMENTAL STATS</codeph>
+        statement.
+      </p>
+
+      <p>
+        The metadata for incremental statistics is handled differently from the original style of statistics:
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            If you have an existing partitioned table for which you have already computed statistics, issuing
+            <codeph>COMPUTE INCREMENTAL STATS</codeph> without a partition clause causes Impala to rescan the
+            entire table. Once the incremental statistics are computed, any future <codeph>COMPUTE INCREMENTAL
+            STATS</codeph> statements only scan any new partitions and any partitions where you performed
+            <codeph>DROP INCREMENTAL STATS</codeph>.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            The <codeph>SHOW TABLE STATS</codeph> and <codeph>SHOW PARTITIONS</codeph> statements now include an
+            additional column showing whether incremental statistics are available for each column. A partition
+            could already be covered by the original type of statistics based on a prior <codeph>COMPUTE
+            STATS</codeph> statement, as indicated by a value other than <codeph>-1</codeph> under the
+            <codeph>#Rows</codeph> column. Impala query planning uses either kind of statistics when available.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            <codeph>COMPUTE INCREMENTAL STATS</codeph> takes more time than <codeph>COMPUTE STATS</codeph> for the
+            same volume of data. Therefore it is most suitable for tables with large data volume where new
+            partitions are added frequently, making it impractical to run a full <codeph>COMPUTE STATS</codeph>
+            operation for each new partition. For unpartitioned tables, or partitioned tables that are loaded once
+            and not updated with new partitions, use the original <codeph>COMPUTE STATS</codeph> syntax.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            <codeph>COMPUTE INCREMENTAL STATS</codeph> uses some memory in the <cmdname>catalogd</cmdname> process,
+            proportional to the number of partitions and number of columns in the applicable table. The memory
+            overhead is approximately 400 bytes for each column in each partition. This memory is reserved in the
+            <cmdname>catalogd</cmdname> daemon, the <cmdname>statestored</cmdname> daemon, and in each instance of
+            the <cmdname>impalad</cmdname> daemon.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            In cases where new files are added to an existing partition, issue a <codeph>REFRESH</codeph> statement
+            for the table, followed by a <codeph>DROP INCREMENTAL STATS</codeph> and <codeph>COMPUTE INCREMENTAL
+            STATS</codeph> sequence for the changed partition.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            The <codeph>DROP INCREMENTAL STATS</codeph> statement operates only on a single partition at a time. To
+            remove statistics (whether incremental or not) from all partitions of a table, issue a <codeph>DROP
+            STATS</codeph> statement with no <codeph>INCREMENTAL</codeph> or <codeph>PARTITION</codeph> clauses.
+          </p>
+        </li>
+      </ul>
+
+      <p>
+        The following considerations apply to incremental statistics when the structure of an existing table is
+        changed (known as <term>schema evolution</term>):
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            If you use an <codeph>ALTER TABLE</codeph> statement to drop a column, the existing statistics remain
+            valid and <codeph>COMPUTE INCREMENTAL STATS</codeph> does not rescan any partitions.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            If you use an <codeph>ALTER TABLE</codeph> statement to add a column, Impala rescans all partitions and
+            fills in the appropriate column-level values the next time you run <codeph>COMPUTE INCREMENTAL
+            STATS</codeph>.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            If you use an <codeph>ALTER TABLE</codeph> statement to change the data type of a column, Impala
+            rescans all partitions and fills in the appropriate column-level values the next time you run
+            <codeph>COMPUTE INCREMENTAL STATS</codeph>.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            If you use an <codeph>ALTER TABLE</codeph> statement to change the file format of a table, the existing
+            statistics remain valid and a subsequent <codeph>COMPUTE INCREMENTAL STATS</codeph> does not rescan any
+            partitions.
+          </p>
+        </li>
+      </ul>
+
+      <p>
+        See <xref href="impala_compute_stats.xml#compute_stats"/> and
+        <xref href="impala_drop_stats.xml#drop_stats"/> for syntax details.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="perf_stats_computing">
+    <title>Generating Table and Column Statistics (COMPUTE STATS Statement)</title>
+    <conbody>
+
+      <p>
+        To gather table statistics after loading data into a table or partition, you typically use the
+        <codeph>COMPUTE STATS</codeph> statement. This statement is available in Impala 1.2.2 and higher.
+        It gathers both table statistics and column statistics for all columns in a single operation.
+        For large partitioned tables, where you frequently need to update statistics and it is impractical
+        to scan the entire table each time, use the syntax <codeph>COMPUTE INCREMENTAL STATS</codeph>,
+        which is available in <keyword keyref="impala21_full"/> and higher.
+      </p>
+
+      <p>
+        If you use Hive as part of your ETL workflow, you can also use Hive to generate table and
+        column statistics. You might need to do extra configuration within Hive itself, the metastore,
+        or even set up a separate database to hold Hive-generated statistics. You might need to run
+        multiple statements to generate all the necessary statistics. Therefore, prefer the
+        Impala <codeph>COMPUTE STATS</codeph> statement where that technique is practical.
+        For details about collecting statistics through Hive, see
+        <xref href="https://cwiki.apache.org/confluence/display/Hive/StatsDev" scope="external" format="html">the Hive wiki</xref>.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/hive_column_stats_caveat"/>
+
+<!-- Commenting out over-detailed Hive instructions as part of stats reorg.
+        <li>
+          Issue an <codeph>ANALYZE TABLE</codeph> statement in Hive, for the entire table or a specific partition.
+<codeblock>ANALYZE TABLE <varname>tablename</varname> [PARTITION(<varname>partcol1</varname>[=<varname>val1</varname>], <varname>partcol2</varname>[=<varname>val2</varname>], ...)] COMPUTE STATISTICS [NOSCAN];</codeblock>
+          For example, to gather statistics for a non-partitioned table:
+<codeblock>ANALYZE TABLE customer COMPUTE STATISTICS;</codeblock>
+          To gather statistics for a <codeph>store</codeph> table partitioned by state and city, and both of its
+          partitions:
+<codeblock>ANALYZE TABLE store PARTITION(s_state, s_county) COMPUTE STATISTICS;</codeblock>
+          To gather statistics for the <codeph>store</codeph> table and only the partitions for California:
+<codeblock>ANALYZE TABLE store PARTITION(s_state='CA', s_county) COMPUTE STATISTICS;</codeblock>
+        </li>
+
+        <li>
+          Load the data through the <codeph>INSERT OVERWRITE</codeph> statement in Hive, while the Hive setting
+          <b>hive.stats.autogather</b> is enabled.
+        </li>
+
+      </ul>
+-->
+
+      <p rev="2.0.1">
+<!-- Additional info as a result of IMPALA-1420 -->
+<!-- Keep checking if https://issues.apache.org/jira/browse/HIVE-8648 ever gets fixed and when that fix makes it into a CDH release. -->
+        For your very largest tables, you might find that <codeph>COMPUTE STATS</codeph> or even <codeph>COMPUTE INCREMENTAL STATS</codeph>
+        take so long to scan the data that it is impractical to use them regularly. In such a case, after adding a partition or inserting new data,
+        you can update just the number of rows property through an <codeph>ALTER TABLE</codeph> statement.
+        See <xref href="impala_perf_stats.xml#perf_table_stats_manual"/> for details.
+        Because the column statistics might be left in a stale state, do not use this technique as a replacement
+        for <codeph>COMPUTE STATS</codeph>. Only use this technique if all other means of collecting statistics are impractical, or as a
+        low-overhead operation that you run in between periodic <codeph>COMPUTE STATS</codeph> or <codeph>COMPUTE INCREMENTAL STATS</codeph> operations.
+      </p>
+
+    </conbody>
+  </concept>
+
+  <concept rev="2.1.0" id="perf_stats_checking">
+
+    <title>Detecting Missing Statistics</title>
+
+    <conbody>
+
+      <p>
+        You can check whether a specific table has statistics using the <codeph>SHOW TABLE STATS</codeph> statement
+        (for any table) or the <codeph>SHOW PARTITIONS</codeph> statement (for a partitioned table). Both
+        statements display the same information. If a table or a partition does not have any statistics, the
+        <codeph>#Rows</codeph> field contains <codeph>-1</codeph>. Once you compute statistics for the table or
+        partition, the <codeph>#Rows</codeph> field changes to an accurate value.
+      </p>
+
+      <p>
+        The following example shows a table that initially does not have any statistics. The <codeph>SHOW TABLE
+        STATS</codeph> statement displays different values for <codeph>#Rows</codeph> before and after the
+        <codeph>COMPUTE STATS</codeph> operation.
+      </p>
+
+<codeblock>[localhost:21000] &gt; create table no_stats (x int);
+[localhost:21000] &gt; show table stats no_stats;
++-------+--------+------+--------------+--------+-------------------+
+| #Rows | #Files | Size | Bytes Cached | Format | Incremental stats |
++-------+--------+------+--------------+--------+-------------------+
+| -1    | 0      | 0B   | NOT CACHED   | TEXT   | false             |
++-------+--------+------+--------------+--------+-------------------+
+[localhost:21000] &gt; compute stats no_stats;
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 1 partition(s) and 1 column(s). |
++-----------------------------------------+
+[localhost:21000] &gt; show table stats no_stats;
++-------+--------+------+--------------+--------+-------------------+
+| #Rows | #Files | Size | Bytes Cached | Format | Incremental stats |
++-------+--------+------+--------------+--------+-------------------+
+| 0     | 0      | 0B   | NOT CACHED   | TEXT   | false             |
++-------+--------+------+--------------+--------+-------------------+
+</codeblock>
+
+      <p>
+        The following example shows a similar progression with a partitioned table. Initially,
+        <codeph>#Rows</codeph> is <codeph>-1</codeph>. After a <codeph>COMPUTE STATS</codeph> operation,
+        <codeph>#Rows</codeph> changes to an accurate value. Any newly added partition starts with no statistics,
+        meaning that you must collect statistics after adding a new partition.
+      </p>
+
+<codeblock>[localhost:21000] &gt; create table no_stats_partitioned (x int) partitioned by (year smallint);
+[localhost:21000] &gt; show table stats no_stats_partitioned;
++-------+-------+--------+------+--------------+--------+-------------------+
+| year  | #Rows | #Files | Size | Bytes Cached | Format | Incremental stats |
++-------+-------+--------+------+--------------+--------+-------------------+
+| Total | -1    | 0      | 0B   | 0B           |        |                   |
++-------+-------+--------+------+--------------+--------+-------------------+
+[localhost:21000] &gt; show partitions no_stats_partitioned;
++-------+-------+--------+------+--------------+--------+-------------------+
+| year  | #Rows | #Files | Size | Bytes Cached | Format | Incremental stats |
++-------+-------+--------+------+--------------+--------+-------------------+
+| Total | -1    | 0      | 0B   | 0B           |        |                   |
++-------+-------+--------+------+--------------+--------+-------------------+
+[localhost:21000] &gt; alter table no_stats_partitioned add partition (year=2013);
+[localhost:21000] &gt; compute stats no_stats_partitioned;
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 1 partition(s) and 1 column(s). |
++-----------------------------------------+
+[localhost:21000] &gt; alter table no_stats_partitioned add partition (year=2014);
+[localhost:21000] &gt; show partitions no_stats_partitioned;
++-------+-------+--------+------+--------------+--------+-------------------+
+| year  | #Rows | #Files | Size | Bytes Cached | Format | Incremental stats |
++-------+-------+--------+------+--------------+--------+-------------------+
+| 2013  | 0     | 0      | 0B   | NOT CACHED   | TEXT   | false             |
+| 2014  | -1    | 0      | 0B   | NOT CACHED   | TEXT   | false             |
+| Total | 0     | 0      | 0B   | 0B           |        |                   |
++-------+-------+--------+------+--------------+--------+-------------------+
+</codeblock>
+
+      <note>
+        Because the default <codeph>COMPUTE STATS</codeph> statement creates and updates statistics for all
+        partitions in a table, if you expect to frequently add new partitions, use the <codeph>COMPUTE INCREMENTAL
+        STATS</codeph> syntax instead, which lets you compute stats for a single specified partition, or only for
+        those partitions that do not already have incremental stats.
+      </note>
+
+      <p>
+        If checking each individual table is impractical, due to a large number of tables or views that hide the
+        underlying base tables, you can also check for missing statistics for a particular query. Use the
+        <codeph>EXPLAIN</codeph> statement to preview query efficiency before actually running the query. Use the
+        query profile output available through the <codeph>PROFILE</codeph> command in
+        <cmdname>impala-shell</cmdname> or the web UI to verify query execution and timing after running the query.
+        Both the <codeph>EXPLAIN</codeph> plan and the <codeph>PROFILE</codeph> output display a warning if any
+        tables or partitions involved in the query do not have statistics.
+      </p>
+
+<codeblock>[localhost:21000] &gt; create table no_stats (x int);
+[localhost:21000] &gt; explain select count(*) from no_stats;
++------------------------------------------------------------------------------------+
+| Explain String                                                                     |
++------------------------------------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=10.00MB VCores=1                           |
+| WARNING: The following tables are missing relevant table and/or column statistics. |
+| incremental_stats.no_stats                                                         |
+|                                                                                    |
+| 03:AGGREGATE [FINALIZE]                                                            |
+| |  output: count:merge(*)                                                          |
+| |                                                                                  |
+| 02:EXCHANGE [UNPARTITIONED]                                                        |
+| |                                                                                  |
+| 01:AGGREGATE                                                                       |
+| |  output: count(*)                                                                |
+| |                                                                                  |
+| 00:SCAN HDFS [incremental_stats.no_stats]                                          |
+|    partitions=1/1 files=0 size=0B                                                  |
++------------------------------------------------------------------------------------+
+</codeblock>
+
+      <p>
+        Because Impala uses the <term>partition pruning</term> technique when possible to only evaluate certain
+        partitions, if you have a partitioned table with statistics for some partitions and not others, whether or
+        not the <codeph>EXPLAIN</codeph> statement shows the warning depends on the actual partitions used by the
+        query. For example, you might see warnings or not for different queries against the same table:
+      </p>
+
+<codeblock>-- No warning because all the partitions for the year 2012 have stats.
+EXPLAIN SELECT ... FROM t1 WHERE year = 2012;
+
+-- Missing stats warning because one or more partitions in this range
+-- do not have stats.
+EXPLAIN SELECT ... FROM t1 WHERE year BETWEEN 2006 AND 2009;
+</codeblock>
+
+      <p>
+        To confirm if any partitions at all in the table are missing statistics, you might explain a query that
+        scans the entire table, such as <codeph>SELECT COUNT(*) FROM <varname>table_name</varname></codeph>.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept rev="2.1.0" id="perf_stats_collecting">
+
+    <title>Keeping Statistics Up to Date</title>
+
+    <conbody>
+
+      <p>
+        When the contents of a table or partition change significantly, recompute the stats for the relevant table
+        or partition. The degree of change that qualifies as <q>significant</q> varies, depending on the absolute
+        and relative sizes of the tables. Typically, if you add more than 30% more data to a table, it is
+        worthwhile to recompute stats, because the differences in number of rows and number of distinct values
+        might cause Impala to choose a different join order when that table is used in join queries. This guideline
+        is most important for the largest tables. For example, adding 30% new data to a table containing 1 TB has a
+        greater effect on join order than adding 30% to a table containing only a few megabytes, and the larger
+        table has a greater effect on query performance if Impala chooses a suboptimal join order as a result of
+        outdated statistics.
+      </p>
+
+      <p>
+        If you reload a complete new set of data for a table, but the number of rows and number of distinct values
+        for each column is relatively unchanged from before, you do not need to recompute stats for the table.
+      </p>
+
+      <p>
+        If the statistics for a table are out of date, and the table's large size makes it impractical to recompute
+        new stats immediately, you can use the <codeph>DROP STATS</codeph> statement to remove the obsolete
+        statistics, making it easier to identify tables that need a new <codeph>COMPUTE STATS</codeph> operation.
+      </p>
+
+      <p>
+        For a large partitioned table, consider using the incremental stats feature available in Impala 2.1.0 and
+        higher, as explained in <xref href="impala_perf_stats.xml#perf_stats_incremental"/>. If you add a new
+        partition to a table, it is worthwhile to recompute incremental stats, because the operation only scans the
+        data for that one new partition.
+      </p>
+    </conbody>
+  </concept>
+
+<!-- Might deserve its own conceptual topic at some point. -->
+
+  <concept audience="Cloudera" rev="1.2.2" id="perf_stats_joins">
+
+    <title>How Statistics Are Used in Join Queries</title>
+
+    <conbody>
+
+      <p></p>
+    </conbody>
+  </concept>
+
+<!-- Might deserve its own conceptual topic at some point. -->
+
+  <concept audience="Cloudera" rev="1.2.2" id="perf_stats_inserts">
+
+    <title>How Statistics Are Used in INSERT Operations</title>
+
+    <conbody>
+
+      <p conref="../shared/impala_common.xml#common/insert_hints"/>
+    </conbody>
+  </concept>
+
+  <concept rev="1.2.2" id="perf_table_stats_manual">
+
+    <title>Setting the NUMROWS Value Manually through ALTER TABLE</title>
+
+    <conbody>
+
+      <p>
+        The most crucial piece of data in all the statistics is the number of rows in the table (for an
+        unpartitioned or partitioned table) and for each partition (for a partitioned table). The <codeph>COMPUTE STATS</codeph>
+        statement always gathers statistics about all columns, as well as overall table statistics. If it is not
+        practical to do a full <codeph>COMPUTE STATS</codeph> or <codeph>COMPUTE INCREMENTAL STATS</codeph>
+        operation after adding a partition or inserting data, or if you can see that Impala would produce a more
+        efficient plan if the number of rows was different, you can manually set the number of rows through an
+        <codeph>ALTER TABLE</codeph> statement:
+      </p>
+
+<codeblock>
+-- Set total number of rows. Applies to both unpartitioned and partitioned tables.
+alter table <varname>table_name</varname> set tblproperties('numRows'='<varname>new_value</varname>', 'STATS_GENERATED_VIA_STATS_TASK'='true');
+
+-- Set total number of rows for a specific partition. Applies to partitioned tables only.
+-- You must specify all the partition key columns in the PARTITION clause.
+alter table <varname>table_name</varname> partition (<varname>keycol1</varname>=<varname>val1</varname>,<varname>keycol2</varname>=<varname>val2</varname>...) set tblproperties('numRows'='<varname>new_value</varname>', 'STATS_GENERATED_VIA_STATS_TASK'='true');
+</codeblock>
+
+      <p>
+        This statement avoids re-scanning any data files. (The requirement to include the <codeph>STATS_GENERATED_VIA_STATS_TASK</codeph> property is relatively new, as a
+        result of the issue <xref href="https://issues.apache.org/jira/browse/HIVE-8648" scope="external" format="html">HIVE-8648</xref>
+        for the Hive metastore.)
+      </p>
+
+<codeblock conref="../shared/impala_common.xml#common/set_numrows_example"/>
+
+      <p>
+        For a partitioned table, update both the per-partition number of rows and the number of rows for the whole
+        table:
+      </p>
+
+<codeblock conref="../shared/impala_common.xml#common/set_numrows_partitioned_example"/>
+
+      <p>
+        In practice, the <codeph>COMPUTE STATS</codeph> statement, or <codeph>COMPUTE INCREMENTAL STATS</codeph>
+        for a partitioned table, should be fast and convenient enough that this technique is only useful for the very
+        largest partitioned tables.
+        <!--
+        It is most useful as a workaround for in case of performance issues where you might adjust the <codeph>numRows</codeph> value higher
+        or lower to produce the ideal join order.
+        -->
+        <!-- Following wording is duplicated from earlier. Consider conref'ing. -->
+        Because the column statistics might be left in a stale state, do not use this technique as a replacement
+        for <codeph>COMPUTE STATS</codeph>. Only use this technique if all other means of collecting statistics are impractical, or as a
+        low-overhead operation that you run in between periodic <codeph>COMPUTE STATS</codeph> or <codeph>COMPUTE INCREMENTAL STATS</codeph> operations.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="perf_column_stats_manual" rev="2.6.0 IMPALA-3369">
+    <title>Setting Column Stats Manually through ALTER TABLE</title>
+    <conbody>
+      <p>
+        In <keyword keyref="impala26_full"/> and higher, you can also use the <codeph>SET COLUMN STATS</codeph>
+        clause of <codeph>ALTER TABLE</codeph> to manually set or change column statistics.
+        Only use this technique in cases where it is impractical to run
+        <codeph>COMPUTE STATS</codeph> or <codeph>COMPUTE INCREMENTAL STATS</codeph>
+        frequently enough to keep up with data changes for a huge table.
+      </p>
+      <p conref="../shared/impala_common.xml#common/set_column_stats_example"/>
+    </conbody>
+  </concept>
+
+  <concept rev="1.2.2" id="perf_stats_examples">
+
+    <title>Examples of Using Table and Column Statistics with Impala</title>
+
+    <conbody>
+
+      <p>
+        The following examples walk through a sequence of <codeph>SHOW TABLE STATS</codeph>, <codeph>SHOW COLUMN
+        STATS</codeph>, <codeph>ALTER TABLE</codeph>, and <codeph>SELECT</codeph> and <codeph>INSERT</codeph>
+        statements to illustrate various aspects of how Impala uses statistics to help optimize queries.
+      </p>
+
+      <p>
+        This example shows table and column statistics for the <codeph>STORE</codeph> column used in the
+        <xref href="http://www.tpc.org/tpcds/" scope="external" format="html">TPC-DS benchmarks for decision
+        support</xref> systems. It is a tiny table holding data for 12 stores. Initially, before any statistics are
+        gathered by a <codeph>COMPUTE STATS</codeph> statement, most of the numeric fields show placeholder values
+        of -1, indicating that the figures are unknown. The figures that are filled in are values that are easily
+        countable or deducible at the physical level, such as the number of files, total data size of the files,
+        and the maximum and average sizes for data types that have a constant size such as <codeph>INT</codeph>,
+        <codeph>FLOAT</codeph>, and <codeph>TIMESTAMP</codeph>.
+      </p>
+
+<codeblock>[localhost:21000] &gt; show table stats store;
++-------+--------+--------+--------+
+| #Rows | #Files | Size   | Format |
++-------+--------+--------+--------+
+| -1    | 1      | 3.08KB | TEXT   |
++-------+--------+--------+--------+
+Returned 1 row(s) in 0.03s
+[localhost:21000] &gt; show column stats store;
++--------------------+-----------+------------------+--------+----------+----------+
+| Column             | Type      | #Distinct Values | #Nulls | Max Size | Avg Size |
++--------------------+-----------+------------------+--------+----------+----------+
+| s_store_sk         | INT       | -1               | -1     | 4        | 4        |
+| s_store_id         | STRING    | -1               | -1     | -1       | -1       |
+| s_rec_start_date   | TIMESTAMP | -1               | -1     | 16       | 16       |
+| s_rec_end_date     | TIMESTAMP | -1               | -1     | 16       | 16       |
+| s_closed_date_sk   | INT       | -1               | -1     | 4        | 4        |
+| s_store_name       | STRING    | -1               | -1     | -1       | -1       |
+| s_number_employees | INT       | -1               | -1     | 4        | 4        |
+| s_floor_space      | INT       | -1               | -1     | 4        | 4        |
+| s_hours            | STRING    | -1               | -1     | -1       | -1       |
+| s_manager          | STRING    | -1               | -1     | -1       | -1       |
+| s_market_id        | INT       | -1               | -1     | 4        | 4        |
+| s_geography_class  | STRING    | -1               | -1     | -1       | -1       |
+| s_market_desc      | STRING    | -1               | -1     | -1       | -1       |
+| s_market_manager   | STRING    | -1               | -1     | -1       | -1       |
+| s_division_id      | INT       | -1               | -1     | 4        | 4        |
+| s_division_name    | STRING    | -1               | -1     | -1       | -1       |
+| s_company_id       | INT       | -1               | -1     | 4        | 4        |
+| s_company_name     | STRING    | -1               | -1     | -1       | -1       |
+| s_street_number    | STRING    | -1               | -1     | -1       | -1       |
+| s_street_name      | STRING    | -1               | -1     | -1       | -1       |
+| s_street_type      | STRING    | -1               | -1     | -1       | -1       |
+| s_suite_number     | STRING    | -1               | -1     | -1       | -1       |
+| s_city             | STRING    | -1               | -1     | -1       | -1       |
+| s_county           | STRING    | -1               | -1     | -1       | -1       |
+| s_state            | STRING    | -1               | -1     | -1       | -1       |
+| s_zip              | STRING    | -1               | -1     | -1       | -1       |
+| s_country          | STRING    | -1               | -1     | -1       | -1       |
+| s_gmt_offset       | FLOAT     | -1               | -1     | 4        | 4        |
+| s_tax_percentage   | FLOAT     | -1               | -1     | 4        | 4        |
++--------------------+-----------+------------------+--------+----------+----------+
+Returned 29 row(s) in 0.04s</codeblock>
+
+      <p>
+        With the Hive <codeph>ANALYZE TABLE</codeph> statement for column statistics, you had to specify each
+        column for which to gather statistics. The Impala <codeph>COMPUTE STATS</codeph> statement automatically
+        gathers statistics for all columns, because it reads through the entire table relatively quickly and can
+        efficiently compute the values for all the columns. This example shows how after running the
+        <codeph>COMPUTE STATS</codeph> statement, statistics are filled in for both the table and all its columns:
+      </p>
+
+<codeblock>[localhost:21000] &gt; compute stats store;
++------------------------------------------+
+| summary                                  |
++------------------------------------------+
+| Updated 1 partition(s) and 29 column(s). |
++------------------------------------------+
+Returned 1 row(s) in 1.88s
+[localhost:21000] &gt; show table stats store;
++-------+--------+--------+--------+
+| #Rows | #Files | Size   | Format |
++-------+--------+--------+--------+
+| 12    | 1      | 3.08KB | TEXT   |
++-------+--------+--------+--------+
+Returned 1 row(s) in 0.02s
+[localhost:21000] &gt; show column stats store;
++--------------------+-----------+------------------+--------+----------+-------------------+
+| Column             | Type      | #Distinct Values | #Nulls | Max Size | Avg Size          |
++--------------------+-----------+------------------+--------+----------+-------------------+
+| s_store_sk         | INT       | 12               | -1     | 4        | 4                 |
+| s_store_id         | STRING    | 6                | -1     | 16       | 16                |
+| s_rec_start_date   | TIMESTAMP | 4                | -1     | 16       | 16                |
+| s_rec_end_date     | TIMESTAMP | 3                | -1     | 16       | 16                |
+| s_closed_date_sk   | INT       | 3                | -1     | 4        | 4                 |
+| s_store_name       | STRING    | 8                | -1     | 5        | 4.25              |
+| s_number_employees | INT       | 9                | -1     | 4        | 4                 |
+| s_floor_space      | INT       | 10               | -1     | 4        | 4                 |
+| s_hours            | STRING    | 2                | -1     | 8        | 7.083300113677979 |
+| s_manager          | STRING    | 7                | -1     | 15       | 12                |
+| s_market_id        | INT       | 7                | -1     | 4        | 4                 |
+| s_geography_class  | STRING    | 1                | -1     | 7        | 7                 |
+| s_market_desc      | STRING    | 10               | -1     | 94       | 55.5              |
+| s_market_manager   | STRING    | 7                | -1     | 16       | 14                |
+| s_division_id      | INT       | 1                | -1     | 4        | 4                 |
+| s_division_name    | STRING    | 1                | -1     | 7        | 7                 |
+| s_company_id       | INT       | 1                | -1     | 4        | 4                 |
+| s_company_name     | STRING    | 1                | -1     | 7        | 7                 |
+| s_street_number    | STRING    | 9                | -1     | 3        | 2.833300113677979 |
+| s_street_name      | STRING    | 12               | -1     | 11       | 6.583300113677979 |
+| s_street_type      | STRING    | 8                | -1     | 9        | 4.833300113677979 |
+| s_suite_number     | STRING    | 11               | -1     | 9        | 8.25              |
+| s_city             | STRING    | 2                | -1     | 8        | 6.5               |
+| s_county           | STRING    | 1                | -1     | 17       | 17                |
+| s_state            | STRING    | 1                | -1     | 2        | 2                 |
+| s_zip              | STRING    | 2                | -1     | 5        | 5                 |
+| s_country          | STRING    | 1                | -1     | 13       | 13                |
+| s_gmt_offset       | FLOAT     | 1                | -1     | 4        | 4                 |
+| s_tax_percentage   | FLOAT     | 5                | -1     | 4        | 4                 |
++--------------------+-----------+------------------+--------+----------+-------------------+
+Returned 29 row(s) in 0.04s</codeblock>
+
+      <p>
+        The following example shows how statistics are represented for a partitioned table. In this case, we have
+        set up a table to hold the world's most trivial census data, a single <codeph>STRING</codeph> field,
+        partitioned by a <codeph>YEAR</codeph> column. The table statistics include a separate entry for each
+        partition, plus final totals for the numeric fields. The column statistics include some easily deducible
+        facts for the partitioning column, such as the number of distinct values (the number of partition
+        subdirectories).
+<!-- and the number of <codeph>NULL</codeph> values (none in this case). -->
+      </p>
+
+<codeblock>localhost:21000] &gt; describe census;
++------+----------+---------+
+| name | type     | comment |
++------+----------+---------+
+| name | string   |         |
+| year | smallint |         |
++------+----------+---------+
+Returned 2 row(s) in 0.02s
+[localhost:21000] &gt; show table stats census;
++-------+-------+--------+------+---------+
+| year  | #Rows | #Files | Size | Format  |
++-------+-------+--------+------+---------+
+| 2000  | -1    | 0      | 0B   | TEXT    |
+| 2004  | -1    | 0      | 0B   | TEXT    |
+| 2008  | -1    | 0      | 0B   | TEXT    |
+| 2010  | -1    | 0      | 0B   | TEXT    |
+| 2011  | 0     | 1      | 22B  | TEXT    |
+| 2012  | -1    | 1      | 22B  | TEXT    |
+| 2013  | -1    | 1      | 231B | PARQUET |
+| Total | 0     | 3      | 275B |         |
++-------+-------+--------+------+---------+
+Returned 8 row(s) in 0.02s
+[localhost:21000] &gt; show column stats census;
++--------+----------+------------------+--------+----------+----------+
+| Column | Type     | #Distinct Values | #Nulls | Max Size | Avg Size |
++--------+----------+------------------+--------+----------+----------+
+| name   | STRING   | -1               | -1     | -1       | -1       |
+| year   | SMALLINT | 7                | -1     | 2        | 2        |
++--------+----------+------------------+--------+----------+----------+
+Returned 2 row(s) in 0.02s</codeblock>
+
+      <p>
+        The following example shows how the statistics are filled in by a <codeph>COMPUTE STATS</codeph> statement
+        in Impala.
+      </p>
+
+<codeblock>[localhost:21000] &gt; compute stats census;
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 3 partition(s) and 1 column(s). |
++-----------------------------------------+
+Returned 1 row(s) in 2.16s
+[localhost:21000] &gt; show table stats census;
++-------+-------+--------+------+---------+
+| year  | #Rows | #Files | Size | Format  |
++-------+-------+--------+------+---------+
+| 2000  | -1    | 0      | 0B   | TEXT    |
+| 2004  | -1    | 0      | 0B   | TEXT    |
+| 2008  | -1    | 0      | 0B   | TEXT    |
+| 2010  | -1    | 0      | 0B   | TEXT    |
+| 2011  | 4     | 1      | 22B  | TEXT    |
+| 2012  | 4     | 1      | 22B  | TEXT    |
+| 2013  | 1     | 1      | 231B | PARQUET |
+| Total | 9     | 3      | 275B |         |
++-------+-------+--------+------+---------+
+Returned 8 row(s) in 0.02s
+[localhost:21000] &gt; show column stats census;
++--------+----------+------------------+--------+----------+----------+
+| Column | Type     | #Distinct Values | #Nulls | Max Size | Avg Size |
++--------+----------+------------------+--------+----------+----------+
+| name   | STRING   | 4                | -1     | 5        | 4.5      |
+| year   | SMALLINT | 7                | -1     | 2        | 2        |
++--------+----------+------------------+--------+----------+----------+
+Returned 2 row(s) in 0.02s</codeblock>
+
+      <p rev="1.4.0">
+        For examples showing how some queries work differently when statistics are available, see
+        <xref href="impala_perf_joins.xml#perf_joins_examples"/>. You can see how Impala executes a query
+        differently in each case by observing the <codeph>EXPLAIN</codeph> output before and after collecting
+        statistics. Measure the before and after query times, and examine the throughput numbers in before and
+        after <codeph>SUMMARY</codeph> or <codeph>PROFILE</codeph> output, to verify how much the improved plan
+        speeds up performance.
+      </p>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_perf_testing.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_perf_testing.xml b/docs/topics/impala_perf_testing.xml
new file mode 100644
index 0000000..d621556
--- /dev/null
+++ b/docs/topics/impala_perf_testing.xml
@@ -0,0 +1,175 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="performance_testing">
+
+  <title>Testing Impala Performance</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Proof of Concept"/>
+      <data name="Category" value="Logs"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+      <!-- Should reorg this topic to use nested topics, not sections. Some keywords like 'logs' buried in section titles. -->
+      <data name="Category" value="Sectionated Pages"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Test to ensure that Impala is configured for optimal performance. If you have installed Impala without
+      Cloudera Manager, complete the processes described in this topic to help ensure a proper configuration. Even
+      if you installed Impala with Cloudera Manager, which automatically applies appropriate configurations, these
+      procedures can be used to verify that Impala is set up correctly.
+    </p>
+
+    <section id="checking_config_performance">
+
+      <title>Checking Impala Configuration Values</title>
+
+      <p>
+        You can inspect Impala configuration values by connecting to your Impala server using a browser.
+      </p>
+
+      <p>
+        <b>To check Impala configuration values:</b>
+      </p>
+
+      <ol>
+        <li>
+          Use a browser to connect to one of the hosts running <codeph>impalad</codeph> in your environment.
+          Connect using an address of the form
+          <codeph>http://<varname>hostname</varname>:<varname>port</varname>/varz</codeph>.
+          <note>
+            In the preceding example, replace <codeph>hostname</codeph> and <codeph>port</codeph> with the name and
+            port of your Impala server. The default port is 25000.
+          </note>
+        </li>
+
+        <li>
+          Review the configured values.
+          <p>
+            For example, to check that your system is configured to use block locality tracking information, you
+            would check that the value for <codeph>dfs.datanode.hdfs-blocks-metadata.enabled</codeph> is
+            <codeph>true</codeph>.
+          </p>
+        </li>
+      </ol>
+
+      <p id="p_31">
+        <b>To check data locality:</b>
+      </p>
+
+      <ol>
+        <li>
+          Execute a query on a dataset that is available across multiple nodes. For example, for a table named
+          <codeph>MyTable</codeph> that has a reasonable chance of being spread across multiple DataNodes:
+<codeblock>[impalad-host:21000] &gt; SELECT COUNT (*) FROM MyTable</codeblock>
+        </li>
+
+        <li>
+          After the query completes, review the contents of the Impala logs. You should find a recent message
+          similar to the following:
+<codeblock>Total remote scan volume = 0</codeblock>
+        </li>
+      </ol>
+
+      <p>
+        The presence of remote scans may indicate <codeph>impalad</codeph> is not running on the correct nodes.
+        This can be because some DataNodes do not have <codeph>impalad</codeph> running or it can be because the
+        <codeph>impalad</codeph> instance that is starting the query is unable to contact one or more of the
+        <codeph>impalad</codeph> instances.
+      </p>
+
+      <p>
+        <b>To understand the causes of this issue:</b>
+      </p>
+
+      <ol>
+        <li>
+          Connect to the debugging web server. By default, this server runs on port 25000. This page lists all
+          <codeph>impalad</codeph> instances running in your cluster. If there are fewer instances than you expect,
+          this often indicates some DataNodes are not running <codeph>impalad</codeph>. Ensure
+          <codeph>impalad</codeph> is started on all DataNodes.
+        </li>
+
+        <li>
+          <!-- To do:
+            There are other references to this tip about the "Impala daemon's hostname" elsewhere. Could reconcile, conref, or link.
+          -->
+          If you are using multi-homed hosts, ensure that the Impala daemon's hostname resolves to the interface on
+          which <codeph>impalad</codeph> is running. The hostname Impala is using is displayed when
+          <codeph>impalad</codeph> starts. To explicitly set the hostname, use the <codeph>--hostname</codeph>�flag.
+        </li>
+
+        <li>
+          Check that <codeph>statestored</codeph> is running as expected. Review the contents of the state store
+          log to ensure all instances of <codeph>impalad</codeph> are listed as having connected to the state
+          store.
+        </li>
+      </ol>
+    </section>
+
+    <section id="checking_config_logs">
+
+      <title>Reviewing Impala Logs</title>
+
+      <p>
+        You can review the contents of the Impala logs for signs that short-circuit reads or block location
+        tracking are not functioning. Before checking logs, execute a simple query against a small HDFS dataset.
+        Completing a query task generates log messages using current settings. Information on starting Impala and
+        executing queries can be found in <xref href="impala_processes.xml#processes"/> and
+        <xref href="impala_impala_shell.xml#impala_shell"/>. Information on logging can be found in
+        <xref href="impala_logging.xml#logging"/>. Log messages and their interpretations are as follows:
+      </p>
+
+      <table>
+        <tgroup cols="2">
+          <colspec colname="1" colwidth="30*"/>
+          <colspec colname="2" colwidth="10*"/>
+          <thead>
+            <row>
+              <entry>
+                Log Message
+              </entry>
+              <entry>
+                Interpretation
+              </entry>
+            </row>
+          </thead>
+          <tbody>
+            <row>
+              <entry>
+                <p>
+<pre>Unknown disk id. This will negatively affect performance. Check your hdfs settings to enable block location metadata
+</pre>
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Tracking block locality is not enabled.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+<pre>Unable to load native-hadoop library for your platform... using builtin-java classes where applicable</pre>
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Native checksumming is not enabled.
+                </p>
+              </entry>
+            </row>
+          </tbody>
+        </tgroup>
+      </table>
+    </section>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_performance.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_performance.xml b/docs/topics/impala_performance.xml
new file mode 100644
index 0000000..e58270e
--- /dev/null
+++ b/docs/topics/impala_performance.xml
@@ -0,0 +1,191 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="performance">
+
+  <title>Tuning Impala for Performance</title>
+  <titlealts audience="PDF"><navtitle>Performance Tuning</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Databases"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Developers"/>
+      <!-- Like Impala Administration, this page has a fair bit of info already, but it could benefit from wiki-style embedded of intro text from those other pages. -->
+      <data name="Category" value="Stub Pages"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      The following sections explain the factors affecting the performance of Impala features, and procedures for
+      tuning, monitoring, and benchmarking Impala queries and other SQL operations.
+    </p>
+
+    <p>
+      This section also describes techniques for maximizing Impala scalability. Scalability is tied to performance:
+      it means that performance remains high as the system workload increases. For example, reducing the disk I/O
+      performed by a query can speed up an individual query, and at the same time improve scalability by making it
+      practical to run more queries simultaneously. Sometimes, an optimization technique improves scalability more
+      than performance. For example, reducing memory usage for a query might not change the query performance much,
+      but might improve scalability by allowing more Impala queries or other kinds of jobs to run at the same time
+      without running out of memory.
+    </p>
+
+    <note>
+      <p>
+        Before starting any performance tuning or benchmarking, make sure your system is configured with all the
+        recommended minimum hardware requirements from <xref href="impala_prereqs.xml#prereqs_hardware"/> and
+        software settings from <xref href="impala_config_performance.xml#config_performance"/>.
+      </p>
+    </note>
+
+    <ul>
+      <li>
+        <xref href="impala_partitioning.xml#partitioning"/>. This technique physically divides the data based on
+        the different values in frequently queried columns, allowing queries to skip reading a large percentage of
+        the data in a table.
+      </li>
+
+      <li>
+        <xref href="impala_perf_joins.xml#perf_joins"/>. Joins are the main class of queries that you can tune at
+        the SQL level, as opposed to changing physical factors such as the file format or the hardware
+        configuration. The related topics <xref href="impala_perf_stats.xml#perf_column_stats"/> and
+        <xref href="impala_perf_stats.xml#perf_table_stats"/> are also important primarily for join performance.
+      </li>
+
+      <li>
+        <xref href="impala_perf_stats.xml#perf_table_stats"/> and
+        <xref href="impala_perf_stats.xml#perf_column_stats"/>. Gathering table and column statistics, using the
+        <codeph>COMPUTE STATS</codeph> statement, helps Impala automatically optimize the performance for join
+        queries, without requiring changes to SQL query statements. (This process is greatly simplified in Impala
+        1.2.2 and higher, because the <codeph>COMPUTE STATS</codeph> statement gathers both kinds of statistics in
+        one operation, and does not require any setup and configuration as was previously necessary for the
+        <codeph>ANALYZE TABLE</codeph> statement in Hive.)
+      </li>
+
+      <li>
+        <xref href="impala_perf_testing.xml#performance_testing"/>. Do some post-setup testing to ensure Impala is
+        using optimal settings for performance, before conducting any benchmark tests.
+      </li>
+
+      <li>
+        <xref href="impala_perf_benchmarking.xml#perf_benchmarks"/>. The configuration and sample data that you use
+        for initial experiments with Impala is often not appropriate for doing performance tests.
+      </li>
+
+      <li>
+        <xref href="impala_perf_resources.xml#mem_limits"/>. The more memory Impala can utilize, the better query
+        performance you can expect. In a cluster running other kinds of workloads as well, you must make tradeoffs
+        to make sure all Hadoop components have enough memory to perform well, so you might cap the memory that
+        Impala can use.
+      </li>
+
+      <li rev="1.2" audience="Cloudera">
+        <xref href="impala_perf_hdfs_caching.xml#hdfs_caching"/>. Impala can use the HDFS caching feature to pin
+        frequently accessed data in memory, reducing disk I/O.
+      </li>
+
+      <li rev="2.2.0">
+        <xref href="impala_s3.xml#s3"/>. Queries against data stored in the Amazon Simple Storage Service (S3)
+        have different performance characteristics than when the data is stored in HDFS.
+      </li>
+    </ul>
+
+    <p outputclass="toc"/>
+
+    <p conref="../shared/impala_common.xml#common/cookbook_blurb"/>
+
+  </conbody>
+
+<!-- Empty/hidden stub sections that might be worth expanding later. -->
+
+  <concept id="perf_network" audience="Cloudera">
+
+    <title>Network Traffic</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_partition_schema" audience="Cloudera">
+
+    <title>Designing Partitioned Tables</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_partition_query" audience="Cloudera">
+
+    <title>Queries on Partitioned Tables</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_monitoring" audience="Cloudera">
+
+    <title>Monitoring Performance through the Impala Web Interface</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Monitoring"/>
+    </metadata>
+  </prolog>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_query_coord" audience="Cloudera">
+
+    <title>Query Coordination</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_bottlenecks" audience="Cloudera">
+
+    <title>Performance Bottlenecks</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_long_queries" audience="Cloudera">
+
+    <title>Managing Long-Running Queries</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_load" audience="Cloudera">
+
+    <title>Performance Considerations for Loading Data</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_file_formats" audience="Cloudera">
+
+    <title>Performance Considerations for File Formats</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_compression" audience="Cloudera">
+
+    <title>Performance Considerations for Compression</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Compression"/>
+    </metadata>
+  </prolog>
+
+    <conbody/>
+  </concept>
+
+  <concept id="perf_codegen" audience="Cloudera">
+
+    <title>Native Code Generation</title>
+
+    <conbody/>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_planning.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_planning.xml b/docs/topics/impala_planning.xml
new file mode 100644
index 0000000..f103ab8
--- /dev/null
+++ b/docs/topics/impala_planning.xml
@@ -0,0 +1,30 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="planning">
+
+  <title>Planning for Impala Deployment</title>
+  <titlealts audience="PDF"><navtitle>Deployment Planning</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Deploying"/>
+      <data name="Category" value="Planning"/>
+      <data name="Category" value="Proof of Concept"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Stub Pages"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">planning</indexterm>
+      Before you set up Impala in production, do some planning to make sure that your hardware setup has sufficient
+      capacity, that your cluster topology is optimal for Impala queries, and that your schema design and ETL
+      processes follow the best practices for Impala.
+    </p>
+
+    <p outputclass="toc"/>
+  </conbody>
+</concept>


[51/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
IMPALA-3398: Add docs to main Impala branch.

These are refugees from doc_prototype. They can be rendered with the
DITA Open Toolkit version 2.3.3 by:

/tmp/dita-ot-2.3.3/bin/dita \
  -i impala.ditamap \
  -f html5 \
  -o $(mktemp -d) \
  -filter impala_html.ditaval

Change-Id: I8861e99adc446f659a04463ca78c79200669484f
Reviewed-on: http://gerrit.cloudera.org:8080/5014
Reviewed-by: John Russell <jr...@cloudera.com>
Tested-by: John Russell <jr...@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/3be0f122
Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/3be0f122
Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/3be0f122

Branch: refs/heads/master
Commit: 3be0f122a5bb5d3549d9240b95a5d29b4a21b1a2
Parents: 46f5ad4
Author: Jim Apple <jb...@cloudera.com>
Authored: Tue Nov 8 16:49:18 2016 -0800
Committer: Jim Apple <jb...@cloudera.com>
Committed: Thu Nov 17 22:38:44 2016 +0000

----------------------------------------------------------------------
 docs/Cloudera-Impala-Release-Notes.ditamap      |   10 +
 docs/generatingImpalaDoc.md                     |   73 +
 docs/images/howto_access_control.png            |  Bin 0 -> 46499 bytes
 .../images/howto_per_node_peak_memory_usage.png |  Bin 0 -> 83464 bytes
 docs/images/howto_show_histogram.png            |  Bin 0 -> 18165 bytes
 .../images/howto_static_server_pools_config.png |  Bin 0 -> 45450 bytes
 docs/images/impala_arch.jpeg                    |  Bin 0 -> 41900 bytes
 docs/images/support_send_diagnostic_data.png    |  Bin 0 -> 34387 bytes
 docs/impala.ditamap                             | 1172 +++
 docs/impala_html.ditaval                        |   21 +
 docs/impala_pdf.ditaval                         |   21 +
 docs/impala_sqlref.ditamap                      |  146 +
 docs/shared/ImpalaVariables.xml                 |   52 +
 docs/shared/impala_common.xml                   | 3690 +++++++++
 docs/topics/impala.xml                          |   77 +
 .../impala_abort_on_default_limit_exceeded.xml  |   23 +
 docs/topics/impala_abort_on_error.xml           |   44 +
 docs/topics/impala_admin.xml                    |   60 +
 docs/topics/impala_admission.xml                |  947 +++
 docs/topics/impala_aggregate_functions.xml      |   33 +
 docs/topics/impala_aliases.xml                  |   87 +
 .../topics/impala_allow_unsupported_formats.xml |   31 +
 docs/topics/impala_alter_function.xml           |   21 +
 docs/topics/impala_alter_table.xml              |  806 ++
 docs/topics/impala_alter_view.xml               |   86 +
 docs/topics/impala_analytic_functions.xml       | 1739 +++++
 docs/topics/impala_appx_count_distinct.xml      |   81 +
 docs/topics/impala_appx_median.xml              |  124 +
 docs/topics/impala_array.xml                    |  269 +
 docs/topics/impala_auditing.xml                 |  260 +
 docs/topics/impala_authentication.xml           |   39 +
 docs/topics/impala_authorization.xml            | 1621 ++++
 docs/topics/impala_avg.xml                      |  225 +
 docs/topics/impala_avro.xml                     |  556 ++
 docs/topics/impala_batch_size.xml               |   38 +
 docs/topics/impala_bigint.xml                   |  102 +
 docs/topics/impala_bit_functions.xml            |  794 ++
 docs/topics/impala_boolean.xml                  |  154 +
 docs/topics/impala_breakpad.xml                 |  256 +
 docs/topics/impala_cdh.xml                      |   25 +
 docs/topics/impala_char.xml                     |  278 +
 docs/topics/impala_cluster_sizing.xml           |  353 +
 docs/topics/impala_cm_installation.xml          |   56 +
 docs/topics/impala_comments.xml                 |   53 +
 docs/topics/impala_complex_types.xml            | 2737 +++++++
 docs/topics/impala_components.xml               |  180 +
 docs/topics/impala_compression_codec.xml        |   98 +
 docs/topics/impala_compute_stats.xml            |  432 ++
 docs/topics/impala_concepts.xml                 |  296 +
 docs/topics/impala_conditional_functions.xml    |  443 ++
 docs/topics/impala_config.xml                   |   57 +
 docs/topics/impala_config_options.xml           |  593 ++
 docs/topics/impala_config_performance.xml       |  179 +
 docs/topics/impala_connecting.xml               |  202 +
 docs/topics/impala_conversion_functions.xml     |  758 ++
 docs/topics/impala_count.xml                    |  236 +
 docs/topics/impala_create_data_source.xml       |   35 +
 docs/topics/impala_create_database.xml          |  137 +
 docs/topics/impala_create_function.xml          |  492 ++
 docs/topics/impala_create_role.xml              |   70 +
 docs/topics/impala_create_table.xml             |  832 ++
 docs/topics/impala_create_view.xml              |  139 +
 docs/topics/impala_data_sources.xml             |   22 +
 docs/topics/impala_databases.xml                |   65 +
 docs/topics/impala_datatypes.xml                |   43 +
 docs/topics/impala_date.xml                     |  104 +
 docs/topics/impala_datetime_functions.xml       | 2482 ++++++
 docs/topics/impala_ddl.xml                      |  150 +
 docs/topics/impala_debug_action.xml             |   33 +
 docs/topics/impala_decimal.xml                  |  817 ++
 docs/topics/impala_default_order_by_limit.xml   |   37 +
 docs/topics/impala_delegation.xml               |   88 +
 docs/topics/impala_delete.xml                   |   65 +
 docs/topics/impala_describe.xml                 |  689 ++
 docs/topics/impala_development.xml              |  231 +
 docs/topics/impala_disable_cached_reads.xml     |   36 +
 docs/topics/impala_disable_codegen.xml          |   38 +
 docs/topics/impala_disable_outermost_topn.xml   |   29 +
 .../impala_disable_row_runtime_filtering.xml    |   65 +
 ...impala_disable_streaming_preaggregations.xml |   45 +
 docs/topics/impala_disable_unsafe_spills.xml    |   53 +
 docs/topics/impala_disk_space.xml               |  129 +
 docs/topics/impala_distinct.xml                 |   61 +
 docs/topics/impala_dml.xml                      |   91 +
 docs/topics/impala_double.xml                   |  100 +
 docs/topics/impala_drop_data_source.xml         |   35 +
 docs/topics/impala_drop_database.xml            |  130 +
 docs/topics/impala_drop_function.xml            |  127 +
 docs/topics/impala_drop_role.xml                |   71 +
 docs/topics/impala_drop_stats.xml               |  279 +
 docs/topics/impala_drop_table.xml               |  150 +
 docs/topics/impala_drop_view.xml                |   49 +
 docs/topics/impala_errata.xml                   | 1378 ++++
 .../impala_exec_single_node_rows_threshold.xml  |   96 +
 docs/topics/impala_explain.xml                  |  228 +
 docs/topics/impala_explain_level.xml            |  350 +
 docs/topics/impala_explain_plan.xml             |  568 ++
 docs/topics/impala_faq.xml                      | 1877 +++++
 docs/topics/impala_faq_base.xml                 |   24 +
 docs/topics/impala_features.xml                 |   21 +
 docs/topics/impala_file_formats.xml             |  270 +
 docs/topics/impala_fixed_issues.xml             | 7092 ++++++++++++++++++
 docs/topics/impala_float.xml                    |   94 +
 docs/topics/impala_functions.xml                |  162 +
 docs/topics/impala_functions_overview.xml       |  116 +
 docs/topics/impala_glossary.xml                 |  834 ++
 docs/topics/impala_grant.xml                    |  122 +
 docs/topics/impala_group_by.xml                 |  139 +
 docs/topics/impala_group_concat.xml             |  135 +
 docs/topics/impala_hadoop.xml                   |  165 +
 docs/topics/impala_having.xml                   |   44 +
 docs/topics/impala_hbase.xml                    |  904 +++
 docs/topics/impala_hbase_cache_blocks.xml       |   40 +
 docs/topics/impala_hbase_caching.xml            |   45 +
 docs/topics/impala_hints.xml                    |  249 +
 docs/topics/impala_howto_rm.xml                 |  420 ++
 docs/topics/impala_identifiers.xml              |  114 +
 docs/topics/impala_impala_shell.xml             |  101 +
 docs/topics/impala_incompatible_changes.xml     | 1519 ++++
 docs/topics/impala_insert.xml                   |  689 ++
 docs/topics/impala_install.xml                  |  106 +
 docs/topics/impala_int.xml                      |   95 +
 docs/topics/impala_intro.xml                    |   81 +
 docs/topics/impala_invalidate_metadata.xml      |  233 +
 docs/topics/impala_isilon.xml                   |  114 +
 docs/topics/impala_jdbc.xml                     |  387 +
 docs/topics/impala_joins.xml                    |  534 ++
 docs/topics/impala_kerberos.xml                 |  363 +
 docs/topics/impala_known_issues.xml             | 1812 +++++
 docs/topics/impala_kudu.xml                     |  167 +
 docs/topics/impala_langref.xml                  |   74 +
 docs/topics/impala_langref_sql.xml              |   35 +
 docs/topics/impala_langref_unsupported.xml      |  312 +
 docs/topics/impala_ldap.xml                     |  354 +
 docs/topics/impala_limit.xml                    |  151 +
 docs/topics/impala_lineage.xml                  |  113 +
 docs/topics/impala_literals.xml                 |  384 +
 docs/topics/impala_live_progress.xml            |   86 +
 docs/topics/impala_live_summary.xml             |  211 +
 docs/topics/impala_load_data.xml                |  256 +
 docs/topics/impala_logging.xml                  |  471 ++
 docs/topics/impala_map.xml                      |  267 +
 docs/topics/impala_math_functions.xml           | 1389 ++++
 docs/topics/impala_max.xml                      |  198 +
 docs/topics/impala_max_block_mgr_memory.xml     |   30 +
 docs/topics/impala_max_errors.xml               |   47 +
 docs/topics/impala_max_io_buffers.xml           |   31 +
 docs/topics/impala_max_num_runtime_filters.xml  |   61 +
 docs/topics/impala_max_scan_range_length.xml    |   55 +
 docs/topics/impala_mem_limit.xml                |  212 +
 docs/topics/impala_min.xml                      |  197 +
 docs/topics/impala_misc_functions.xml           |  196 +
 docs/topics/impala_mixed_security.xml           |   46 +
 docs/topics/impala_ndv.xml                      |  135 +
 docs/topics/impala_new_features.xml             | 4015 ++++++++++
 docs/topics/impala_noncm_installation.xml       |  171 +
 docs/topics/impala_num_nodes.xml                |   60 +
 docs/topics/impala_num_scanner_threads.xml      |   35 +
 docs/topics/impala_odbc.xml                     |  203 +
 docs/topics/impala_offset.xml                   |   66 +
 docs/topics/impala_operators.xml                | 1592 ++++
 .../impala_optimize_partition_key_scans.xml     |  180 +
 docs/topics/impala_order_by.xml                 |  318 +
 docs/topics/impala_parquet.xml                  | 1160 +++
 .../impala_parquet_annotate_strings_utf8.xml    |   50 +
 .../topics/impala_parquet_compression_codec.xml |   26 +
 ...mpala_parquet_fallback_schema_resolution.xml |   49 +
 docs/topics/impala_parquet_file_size.xml        |   86 +
 docs/topics/impala_partitioning.xml             |  582 ++
 docs/topics/impala_perf_benchmarking.xml        |   36 +
 docs/topics/impala_perf_cookbook.xml            |  269 +
 docs/topics/impala_perf_ddl.xml                 |   42 +
 docs/topics/impala_perf_hdfs_caching.xml        |  609 ++
 docs/topics/impala_perf_joins.xml               |  506 ++
 docs/topics/impala_perf_resources.xml           |   60 +
 docs/topics/impala_perf_skew.xml                |  150 +
 docs/topics/impala_perf_stats.xml               | 1031 +++
 docs/topics/impala_perf_testing.xml             |  175 +
 docs/topics/impala_performance.xml              |  191 +
 docs/topics/impala_planning.xml                 |   30 +
 docs/topics/impala_porting.xml                  |  623 ++
 docs/topics/impala_ports.xml                    |  440 ++
 docs/topics/impala_prefetch_mode.xml            |   49 +
 docs/topics/impala_prereqs.xml                  |  357 +
 docs/topics/impala_processes.xml                |  134 +
 docs/topics/impala_proxy.xml                    |  635 ++
 docs/topics/impala_query_lifetime.xml           |   31 +
 docs/topics/impala_query_options.xml            |   77 +
 docs/topics/impala_query_timeout_s.xml          |   56 +
 docs/topics/impala_rcfile.xml                   |  244 +
 docs/topics/impala_real.xml                     |   46 +
 docs/topics/impala_refresh.xml                  |  324 +
 docs/topics/impala_release_notes.xml            |   17 +
 docs/topics/impala_relnotes.xml                 |   34 +
 docs/topics/impala_replica_preference.xml       |   48 +
 docs/topics/impala_request_pool.xml             |   43 +
 .../impala_reservation_request_timeout.xml      |   38 +
 docs/topics/impala_reserved_words.xml           |  365 +
 docs/topics/impala_resource_management.xml      |  338 +
 docs/topics/impala_revoke.xml                   |  101 +
 docs/topics/impala_rm_initial_mem.xml           |   29 +
 .../topics/impala_runtime_bloom_filter_size.xml |   93 +
 docs/topics/impala_runtime_filter_max_size.xml  |   51 +
 docs/topics/impala_runtime_filter_min_size.xml  |   51 +
 docs/topics/impala_runtime_filter_mode.xml      |   77 +
 .../impala_runtime_filter_wait_time_ms.xml      |   47 +
 docs/topics/impala_runtime_filtering.xml        |  506 ++
 docs/topics/impala_s3.xml                       |  796 ++
 docs/topics/impala_s3_skip_insert_staging.xml   |   77 +
 docs/topics/impala_scalability.xml              |  833 ++
 .../impala_scan_node_codegen_threshold.xml      |   75 +
 docs/topics/impala_schedule_random_replica.xml  |   80 +
 docs/topics/impala_schema_design.xml            |  222 +
 docs/topics/impala_schema_objects.xml           |   57 +
 docs/topics/impala_security.xml                 |  123 +
 docs/topics/impala_security_files.xml           |   67 +
 docs/topics/impala_security_guidelines.xml      |  108 +
 docs/topics/impala_security_install.xml         |   24 +
 docs/topics/impala_security_metastore.xml       |   40 +
 docs/topics/impala_security_webui.xml           |   66 +
 docs/topics/impala_select.xml                   |  206 +
 docs/topics/impala_seq_compression_mode.xml     |   29 +
 docs/topics/impala_seqfile.xml                  |  239 +
 docs/topics/impala_set.xml                      |  192 +
 docs/topics/impala_shell_commands.xml           |  399 +
 docs/topics/impala_shell_options.xml            |  580 ++
 docs/topics/impala_shell_running_commands.xml   |  265 +
 docs/topics/impala_show.xml                     | 1264 ++++
 docs/topics/impala_smallint.xml                 |  101 +
 docs/topics/impala_ssl.xml                      |  267 +
 docs/topics/impala_stddev.xml                   |  118 +
 docs/topics/impala_string.xml                   |  180 +
 docs/topics/impala_string_functions.xml         |  926 +++
 docs/topics/impala_struct.xml                   |  433 ++
 docs/topics/impala_subqueries.xml               |  320 +
 docs/topics/impala_sum.xml                      |  239 +
 docs/topics/impala_support_start_over.xml       |   30 +
 docs/topics/impala_sync_ddl.xml                 |   59 +
 docs/topics/impala_tables.xml                   |  258 +
 docs/topics/impala_timeouts.xml                 |  172 +
 docs/topics/impala_timestamp.xml                |  445 ++
 docs/topics/impala_tinyint.xml                  |  103 +
 docs/topics/impala_trouble_bad_results.xml      |   25 +
 docs/topics/impala_trouble_memory.xml           |   25 +
 docs/topics/impala_trouble_query_fail.xml       |   24 +
 docs/topics/impala_trouble_sql.xml              |   25 +
 docs/topics/impala_trouble_startup.xml          |   25 +
 docs/topics/impala_troubleshooting.xml          |  447 ++
 docs/topics/impala_truncate_table.xml           |  184 +
 docs/topics/impala_tutorial.xml                 | 2500 ++++++
 docs/topics/impala_txtfile.xml                  |  812 ++
 docs/topics/impala_udf.xml                      | 1933 +++++
 docs/topics/impala_union.xml                    |  152 +
 docs/topics/impala_update.xml                   |   65 +
 docs/topics/impala_upgrading.xml                |  378 +
 docs/topics/impala_upsert.xml                   |   81 +
 docs/topics/impala_use.xml                      |   79 +
 docs/topics/impala_v_cpu_cores.xml              |   39 +
 docs/topics/impala_varchar.xml                  |  217 +
 docs/topics/impala_variance.xml                 |  129 +
 docs/topics/impala_views.xml                    |  187 +
 docs/topics/impala_webui.xml                    |  650 ++
 docs/topics/impala_window_functions.xml         |   23 +
 docs/topics/impala_with.xml                     |   66 +
 264 files changed, 89657 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/Cloudera-Impala-Release-Notes.ditamap
----------------------------------------------------------------------
diff --git a/docs/Cloudera-Impala-Release-Notes.ditamap b/docs/Cloudera-Impala-Release-Notes.ditamap
new file mode 100644
index 0000000..7545b2e
--- /dev/null
+++ b/docs/Cloudera-Impala-Release-Notes.ditamap
@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE map PUBLIC "-//OASIS//DTD DITA Map//EN" "map.dtd">
+<map audience="standalone">
+  <title>Cloudera Impala Release Notes</title>
+  <topicref href="topics/impala_relnotes.xml" audience="HTML standalone"/>
+  <topicref href="topics/impala_new_features.xml"/>
+  <topicref href="topics/impala_incompatible_changes.xml"/>
+  <topicref href="topics/impala_known_issues.xml"/>
+  <topicref href="topics/impala_fixed_issues.xml"/>
+</map>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/generatingImpalaDoc.md
----------------------------------------------------------------------
diff --git a/docs/generatingImpalaDoc.md b/docs/generatingImpalaDoc.md
new file mode 100644
index 0000000..5021e39
--- /dev/null
+++ b/docs/generatingImpalaDoc.md
@@ -0,0 +1,73 @@
+#Generating HTML or a PDF of Apache Impala (Incubating) Documentation
+
+##Prerequisites:
+Make sure that you have a recent version of a Java JDK installed and that your JAVA\_HOME environment variable is set. This procedure has been tested with JDK 1.8.0. See [Setting JAVA\_HOME](#settingjavahome) at the end of these instructions.
+
+* Open a terminal window and run the following commands to get the Impala documentation source files from Git:
+     
+     <pre><code>git clone https://git-wip-us.apache.org/repos/asf/incubator-impala.git/docs
+    cd \<local\_directory\>
+    git checkout doc\_prototype</code></pre>
+    
+    Where <code>doc\_prototype</code> is the branch where Impala documentation source files are uploaded.
+
+* Download the DITA Open Toolkit version 2.3.3 from the DITA Open Toolkit web site:
+   
+   [https://github.com/dita-ot/dita-ot/releases/download/2.3.3/dita-ot-2.3.3.zip] (https://github.com/dita-ot/dita-ot/releases/download/2.3.3/dita-ot-2.3.3.zip)
+   
+  **Note:** A DITA-OT 2.3.3 User Guide is included in the toolkit. Look for <code>userguide.pdf</code> in the <code>doc</code> directory of the toolkit after you extract it. For example, if you extract the toolkit package to the <code>/Users/\<_username_\>/DITA-OT</code> directory on Mac OS, you will find the <code>userguide.pdf</code> at the following location:
+  
+  <code>/Users/\<_username_\>/DITA-OT/doc/userguide.pdf</code>
+
+##To generate HTML or PDF:
+
+1. In the directory where you cloned the Impala documentation files, you will find the following important configuration files in the <code>docs</code> subdirectory. These files are used to convert the XML source you downloaded from the Apache site to PDF and HTML:
+    * <code>impala.ditamap</code>: Tells the DITA Open Toolkit what topics to include in the Impala User/Administration Guide. This guide also includes the Impala SQL Reference.
+    * <code>impala\_sqlref.ditamap</code>: Tells the DITA Open Toolkit what topics to include in the Impala SQL Reference.
+    * <code>impala\_html.ditaval</code>: Further defines what topics to include in the Impala HTML output.
+    * <code>impala\_pdf.ditaval</code>: Further defines what topics to include in the Impala PDF output.
+2. Extract the contents of the DITA-OT package into a directory where you want to generate the HTML or the PDF.
+3. Open a terminal window and navigate to the directory where you extracted the DITA-OT package.
+4.  Run one of the following commands, depending on what you want to generate:
+    * **To generate HTML output of the Impala User and Administration Guide, which includes the Impala SQL Reference, run the following command:**
+    
+        <code>./bin/dita -input \<path\_to\_impala.ditamap\> -format html5 -output \<path\_to\_build\_output\_directory\> -filter \<path\_to\_impala\_html.ditaval\></code>
+     
+    * **To generate PDF output of the Impala User and Administration Guide, which includes the Impala SQL Reference, run the following command:**
+     
+        <code>./bin/dita -input \<path\_to\_impala.ditamap\> -format pdf -output \<path\_to\_build\_output\_directory\> -filter \<path\_to\_impala\_pdf.ditaval\></code>
+        
+    * **To generate HTML output of the Impala SQL Reference, run the following command:**
+     
+        <code>./bin/dita -input \<path\_to\_impala\_sqlref.ditamap\> -format html5 -output \<path\_to\_build\_output\_directory\> -filter \<path\_to\_impala\_html.ditaval\></code>
+     
+    * **To generate PDF output of the Impala SQL Reference, run the following command:**
+     
+        <code>./bin/dita -input \<path\_to\_impala\_sqlref.ditamap\> -format pdf -output \<path\_to\_build\_output\_directory\> -filter \<path\_to\_impala\_pdf.ditaval\></code>
+
+    **Note:** For a description of all command-line options, see the _DITA Open Toolkit User Guide_ in the <code>doc</code> directory of your downloaded DITA Open Toolkit.
+ 
+5. Go to the output directory that you specified in Step 3 to view the HTML or PDF that you generated. If you generated HTML, open the <code>index.html</code> file with a browser to view the output.
+
+<a name="settingjavahome" />
+#Setting JAVA\_HOME
+</a>
+
+Set your JAVA\_HOME environment variable to tell your computer where to find the Java executable file. For example, to set your JAVA\_HOME environment on Mac OS X when you the the 1.8.0\_101 version of the Java Development Kit (JDK) installed and you are using the Bash version 3.2 shell, perform the following steps:
+
+1. Edit your <code>/Users/\<username\>/.bash\_profile</code> file and add the following lines to the end of the file:
+
+    <pre><code>#Set JAVA_HOME
+    JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home
+    export JAVA_HOME;</code></pre>
+    
+   Where <code>jdk1.8.0\_101.jdk</code> is the version of JDK that you have installed. For example, if you have installed <code>jdk1.8.0\_102.jdk</code>, you would use that value instead.
+   
+2. Test to make sure you have set your JAVA\_HOME correctly:
+    * Open a terminal window and type: <code>$JAVA\_HOME/bin/java -version</code>
+    * Press return. If you see something like the following:
+      <pre><code>java version "1.5.0_16"
+      Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b06-284)
+      Java HotSpot (TM) Client VM (build 1.5.0\_16-133, mixed mode, sharing)</code></pre>
+      
+      Then you've successfully set your JAVA\_HOME environment variable to the binary stored in <code>/Library/Java/JavaVirtualMachines/jdk1.8.0\_101.jdk/Contents/Home</code>. 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/images/howto_access_control.png
----------------------------------------------------------------------
diff --git a/docs/images/howto_access_control.png b/docs/images/howto_access_control.png
new file mode 100644
index 0000000..7f4b1bd
Binary files /dev/null and b/docs/images/howto_access_control.png differ

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/images/howto_per_node_peak_memory_usage.png
----------------------------------------------------------------------
diff --git a/docs/images/howto_per_node_peak_memory_usage.png b/docs/images/howto_per_node_peak_memory_usage.png
new file mode 100644
index 0000000..6289a24
Binary files /dev/null and b/docs/images/howto_per_node_peak_memory_usage.png differ

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/images/howto_show_histogram.png
----------------------------------------------------------------------
diff --git a/docs/images/howto_show_histogram.png b/docs/images/howto_show_histogram.png
new file mode 100644
index 0000000..5a02f70
Binary files /dev/null and b/docs/images/howto_show_histogram.png differ

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/images/howto_static_server_pools_config.png
----------------------------------------------------------------------
diff --git a/docs/images/howto_static_server_pools_config.png b/docs/images/howto_static_server_pools_config.png
new file mode 100644
index 0000000..a5fd527
Binary files /dev/null and b/docs/images/howto_static_server_pools_config.png differ

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/images/impala_arch.jpeg
----------------------------------------------------------------------
diff --git a/docs/images/impala_arch.jpeg b/docs/images/impala_arch.jpeg
new file mode 100644
index 0000000..8289469
Binary files /dev/null and b/docs/images/impala_arch.jpeg differ

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/images/support_send_diagnostic_data.png
----------------------------------------------------------------------
diff --git a/docs/images/support_send_diagnostic_data.png b/docs/images/support_send_diagnostic_data.png
new file mode 100644
index 0000000..3ae4bea
Binary files /dev/null and b/docs/images/support_send_diagnostic_data.png differ


[47/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala.xml b/docs/topics/impala.xml
new file mode 100644
index 0000000..8dc7b2a
--- /dev/null
+++ b/docs/topics/impala.xml
@@ -0,0 +1,77 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="about_impala">
+
+  <title>Apache Impala (incubating) - Interactive SQL</title>
+  <titlealts audience="PDF"><navtitle>Impala Guide</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Components"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="SQL"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p conref="../shared/impala_common.xml#common/impala_mission_statement"/>
+
+    <p conref="../shared/impala_common.xml#common/impala_hive_compatibility"/>
+
+    <p conref="../shared/impala_common.xml#common/impala_advantages"/>
+
+    <p outputclass="toc"/>
+
+    <p audience="integrated">
+      <b>Related information throughout the CDH 5 library:</b>
+    </p>
+
+    <p audience="integrated">
+      In CDH 5, the Impala documentation for Release Notes, Installation, Upgrading, and Security has been
+      integrated alongside the corresponding information for other Hadoop components:
+    </p>
+
+<!-- Same list is in impala.xml and Impala FAQs. Conref in both places. -->
+
+    <ul>
+      <li>
+        <xref href="impala_new_features.xml#new_features">New features</xref>
+      </li>
+
+      <li>
+        <xref href="impala_known_issues.xml#known_issues">Known and fixed issues</xref>
+      </li>
+
+      <li>
+        <xref href="impala_incompatible_changes.xml#incompatible_changes">Incompatible changes</xref>
+      </li>
+
+      <li>
+        <xref href="impala_install.xml#install">Installing Impala</xref>
+      </li>
+
+      <li>
+        <xref href="impala_upgrading.xml#upgrading">Upgrading Impala</xref>
+      </li>
+
+      <li>
+        <xref href="impala_config.xml#config">Configuring Impala</xref>
+      </li>
+
+      <li>
+        <xref href="impala_processes.xml#processes">Starting Impala</xref>
+      </li>
+
+      <li>
+        <xref href="impala_security.xml#security">Security for Impala</xref>
+      </li>
+
+      <li>
+        <xref href="http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH-Version-and-Packaging-Information/CDH-Version-and-Packaging-Information.html" scope="external" format="html">CDH
+        Version and Packaging Information</xref>
+      </li>
+    </ul>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_abort_on_default_limit_exceeded.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_abort_on_default_limit_exceeded.xml b/docs/topics/impala_abort_on_default_limit_exceeded.xml
new file mode 100644
index 0000000..5b159a7
--- /dev/null
+++ b/docs/topics/impala_abort_on_default_limit_exceeded.xml
@@ -0,0 +1,23 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="obwl" id="abort_on_default_limit_exceeded">
+
+  <title>ABORT_ON_DEFAULT_LIMIT_EXCEEDED Query Option</title>
+  <titlealts audience="PDF"><navtitle>ABORT_ON_DEFAULT_LIMIT_EXCEEDED</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p conref="../shared/impala_common.xml#common/obwl_query_options"/>
+
+    <p conref="../shared/impala_common.xml#common/type_boolean"/>
+    <p conref="../shared/impala_common.xml#common/default_false_0"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_abort_on_error.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_abort_on_error.xml b/docs/topics/impala_abort_on_error.xml
new file mode 100644
index 0000000..950f8a4
--- /dev/null
+++ b/docs/topics/impala_abort_on_error.xml
@@ -0,0 +1,44 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="abort_on_error">
+
+  <title>ABORT_ON_ERROR Query Option</title>
+  <titlealts audience="PDF"><navtitle>ABORT_ON_ERROR</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">ABORT_ON_ERROR query option</indexterm>
+      When this option is enabled, Impala cancels a query immediately when any of the nodes encounters an error,
+      rather than continuing and possibly returning incomplete results. This option is disabled by default, to help
+      gather maximum diagnostic information when an error occurs, for example, whether the same problem occurred on
+      all nodes or only a single node. Currently, the errors that Impala can skip over involve data corruption,
+      such as a column that contains a string value when expected to contain an integer value.
+    </p>
+
+    <p>
+      To control how much logging Impala does for non-fatal errors when <codeph>ABORT_ON_ERROR</codeph> is turned
+      off, use the <codeph>MAX_ERRORS</codeph> option.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_boolean"/>
+    <p conref="../shared/impala_common.xml#common/default_false_0"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_max_errors.xml#max_errors"/>,
+      <xref href="impala_logging.xml#logging"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_admin.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_admin.xml b/docs/topics/impala_admin.xml
new file mode 100644
index 0000000..3da7d5f
--- /dev/null
+++ b/docs/topics/impala_admin.xml
@@ -0,0 +1,60 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="admin">
+
+  <title>Impala Administration</title>
+  <titlealts audience="PDF"><navtitle>Administration</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Administrators"/>
+      <!-- Although there is a reasonable amount of info on the page, it could be better to use wiki-style embedding instead of linking hither and thither. -->
+      <data name="Category" value="Stub Pages"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      As an administrator, you monitor Impala's use of resources and take action when necessary to keep Impala
+      running smoothly and avoid conflicts with other Hadoop components running on the same cluster. When you
+      detect that an issue has happened or could happen in the future, you reconfigure Impala or other components
+      such as HDFS or even the hardware of the cluster itself to resolve or avoid problems.
+    </p>
+
+    <p outputclass="toc"/>
+
+    <p>
+      <b>Related tasks:</b>
+    </p>
+
+    <p>
+      As an administrator, you can expect to perform installation, upgrade, and configuration tasks for Impala on
+      all machines in a cluster. See <xref href="impala_install.xml#install"/>,
+      <xref href="impala_upgrading.xml#upgrading"/>, and <xref href="impala_config.xml#config"/> for details.
+    </p>
+
+    <p>
+      For security tasks typically performed by administrators, see <xref href="impala_security.xml#security"/>.
+    </p>
+
+    <p>
+      Administrators also decide how to allocate cluster resources so that all Hadoop components can run smoothly
+      together. For Impala, this task primarily involves:
+      <ul>
+        <li>
+          Deciding how many Impala queries can run concurrently and with how much memory, through the admission
+          control feature. See <xref href="impala_admission.xml#admission_control"/> for details.
+        </li>
+
+        <li>
+          Dividing cluster resources such as memory between Impala and other components, using YARN for overall
+          resource management, and Llama to mediate resource requests from Impala to YARN. See
+          <xref href="impala_resource_management.xml#resource_management"/> for details.
+        </li>
+      </ul>
+    </p>
+
+<!-- <p conref="../shared/impala_common.xml#common/impala_mr"/> -->
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_admission.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_admission.xml b/docs/topics/impala_admission.xml
new file mode 100644
index 0000000..ad53823
--- /dev/null
+++ b/docs/topics/impala_admission.xml
@@ -0,0 +1,947 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.3.0" id="admission_control">
+
+  <title>Admission Control and Query Queuing</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Admission Control"/>
+      <data name="Category" value="Resource Management"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p id="admission_control_intro">
+      Admission control is an Impala feature that imposes limits on concurrent SQL queries, to avoid resource usage
+      spikes and out-of-memory conditions on busy CDH clusters.
+      It is a form of <q>throttling</q>.
+      New queries are accepted and executed until
+      certain conditions are met, such as too many queries or too much
+      total memory used across the cluster.
+      When one of these thresholds is reached,
+      incoming queries wait to begin execution. These queries are
+      queued and are admitted (that is, begin executing) when the resources become available.
+    </p>
+    <p>
+      In addition to the threshold values for currently executing queries,
+      you can place limits on the maximum number of queries that are
+      queued (waiting) and a limit on the amount of time they might wait
+      before returning with an error. These queue settings let you ensure that queries do
+      not wait indefinitely, so that you can detect and correct <q>starvation</q> scenarios.
+    </p>
+    <p>
+      Enable this feature if your cluster is
+      underutilized at some times and overutilized at others. Overutilization is indicated by performance
+      bottlenecks and queries being cancelled due to out-of-memory conditions, when those same queries are
+      successful and perform well during times with less concurrent load. Admission control works as a safeguard to
+      avoid out-of-memory conditions during heavy concurrent usage.
+    </p>
+
+    <note conref="../shared/impala_common.xml#common/impala_llama_obsolete"/>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="admission_intro">
+
+    <title>Overview of Impala Admission Control</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Concepts"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        On a busy CDH cluster, you might find there is an optimal number of Impala queries that run concurrently.
+        For example, when the I/O capacity is fully utilized by I/O-intensive queries,
+        you might not find any throughput benefit in running more concurrent queries.
+        By allowing some queries to run at full speed while others wait, rather than having
+        all queries contend for resources and run slowly, admission control can result in higher overall throughput.
+      </p>
+
+      <p>
+        For another example, consider a memory-bound workload such as many large joins or aggregation queries.
+        Each such query could briefly use many gigabytes of memory to process intermediate results.
+        Because Impala by default cancels queries that exceed the specified memory limit,
+        running multiple large-scale queries at once might require
+        re-running some queries that are cancelled. In this case, admission control improves the
+        reliability and stability of the overall workload by only allowing as many concurrent queries
+        as the overall memory of the cluster can accomodate.
+      </p>
+
+      <p>
+        The admission control feature lets you set an upper limit on the number of concurrent Impala
+        queries and on the memory used by those queries. Any additional queries are queued until the earlier ones
+        finish, rather than being cancelled or running slowly and causing contention. As other queries finish, the
+        queued queries are allowed to proceed.
+      </p>
+
+      <p rev="2.5.0">
+        In <keyword keyref="impala25_full"/> and higher, you can specify these limits and thresholds for each
+        pool rather than globally. That way, you can balance the resource usage and throughput
+        between steady well-defined workloads, rare resource-intensive queries, and ad hoc
+        exploratory queries.
+      </p>
+
+      <p>
+        For details on the internal workings of admission control, see
+        <xref href="impala_admission.xml#admission_architecture"/>.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="admission_concurrency">
+    <title>Concurrent Queries and Admission Control</title>
+    <conbody>
+      <p>
+        One way to limit resource usage through admission control is to set an upper limit
+        on the number of concurrent queries. This is the initial technique you might use
+        when you do not have extensive information about memory usage for your workload.
+        This setting can be specified separately for each dynamic resource pool.
+      </p>
+      <p>
+        You can combine this setting with the memory-based approach described in
+        <xref href="impala_admission.xml#admission_memory"/>. If either the maximum number of
+        or the expected memory usage of the concurrent queries is exceeded, subsequent queries
+        are queued until the concurrent workload falls below the threshold again.
+      </p>
+      <p>
+        See
+        <xref audience="integrated" href="cm_mc_resource_pools.xml#concept_xkk_l1d_wr"/><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_resource_pools.html" scope="external" format="html"/>
+        for information about all these dynamic resource
+        pool settings, how to use them together, and how to divide different parts of your workload among
+        different pools.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="admission_memory">
+    <title>Memory Limits and Admission Control</title>
+    <conbody>
+      <p>
+        Each dynamic resource pool can have an upper limit on the cluster-wide memory used by queries executing in that pool.
+        This is the technique to use once you have a stable workload with well-understood memory requirements.
+      </p>
+      <p>
+        Always specify the <uicontrol>Default Query Memory Limit</uicontrol> for the expected maximum amount of RAM
+        that a query might require on each host, which is equivalent to setting the <codeph>MEM_LIMIT</codeph>
+        query option for every query run in that pool. That value affects the execution of each query, preventing it
+        from overallocating memory on each host, and potentially activating the spill-to-disk mechanism or cancelling
+        the query when necessary.
+      </p>
+      <p>
+        Optionally, specify the <uicontrol>Max Memory</uicontrol> setting, a cluster-wide limit that determines
+        how many queries can be safely run concurrently, based on the upper memory limit per host multiplied by the
+        number of Impala nodes in the cluster.
+      </p>
+      <p conref="../shared/impala_common.xml#common/admission_control_mem_limit_interaction"/>
+      <note conref="../shared/impala_common.xml#common/max_memory_default_limit_caveat"/>
+      <p>
+        You can combine the memory-based settings with the upper limit on concurrent queries described in
+        <xref href="impala_admission.xml#admission_concurrency"/>. If either the maximum number of
+        or the expected memory usage of the concurrent queries is exceeded, subsequent queries
+        are queued until the concurrent workload falls below the threshold again.
+      </p>
+      <p>
+        See
+        <xref audience="integrated" href="cm_mc_resource_pools.xml#concept_xkk_l1d_wr"/><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_resource_pools.html" scope="external" format="html"/>
+        for information about all these dynamic resource
+        pool settings, how to use them together, and how to divide different parts of your workload among
+        different pools.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="admission_yarn">
+
+    <title>How Impala Admission Control Relates to Other Resource Management Tools</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Concepts"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        The admission control feature is similar in some ways to the Cloudera Manager
+        static partitioning feature, as well as the YARN resource management framework. These features
+        can be used separately or together. This section describes some similarities and differences, to help you
+        decide which combination of resource management features to use for Impala.
+      </p>
+
+      <p>
+        Admission control is a lightweight, decentralized system that is suitable for workloads consisting
+        primarily of Impala queries and other SQL statements. It sets <q>soft</q> limits that smooth out Impala
+        memory usage during times of heavy load, rather than taking an all-or-nothing approach that cancels jobs
+        that are too resource-intensive.
+      </p>
+
+      <p>
+        Because the admission control system does not interact with other Hadoop workloads such as MapReduce jobs, you
+        might use YARN with static service pools on CDH 5 clusters where resources are shared between
+        Impala and other Hadoop components. This configuration is recommended when using Impala in a
+        <term>multitenant</term> cluster. Devote a percentage of cluster resources to Impala, and allocate another
+        percentage for MapReduce and other batch-style workloads. Let admission control handle the concurrency and
+        memory usage for the Impala work within the cluster, and let YARN manage the work for other components within the
+        cluster. In this scenario, Impala's resources are not managed by YARN.
+      </p>
+
+      <p>
+        The Impala admission control feature uses the same configuration mechanism as the YARN resource manager to map users to
+        pools and authenticate them.
+      </p>
+
+      <p rev="DOCS-648">
+        Although the Impala admission control feature uses a <codeph>fair-scheduler.xml</codeph> configuration file
+        behind the scenes, this file does not depend on which scheduler is used for YARN. You still use this file,
+        and Cloudera Manager can generate it for you, even when YARN is using the capacity scheduler.
+      </p>
+
+    </conbody>
+  </concept>
+
+  <concept id="admission_architecture">
+
+    <title>How Impala Schedules and Enforces Limits on Concurrent Queries</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Concepts"/>
+      <data name="Category" value="Scheduling"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        The admission control system is decentralized, embedded in each Impala daemon and communicating through the
+        statestore mechanism. Although the limits you set for memory usage and number of concurrent queries apply
+        cluster-wide, each Impala daemon makes its own decisions about whether to allow each query to run
+        immediately or to queue it for a less-busy time. These decisions are fast, meaning the admission control
+        mechanism is low-overhead, but might be imprecise during times of heavy load across many coordinators. There could be times when the
+        more queries were queued (in aggregate across the cluster) than the specified limit, or when number of admitted queries
+        exceeds the expected number. Thus, you typically err on the
+        high side for the size of the queue, because there is not a big penalty for having a large number of queued
+        queries; and you typically err on the low side for configuring memory resources, to leave some headroom in case more
+        queries are admitted than expected, without running out of memory and being cancelled as a result.
+      </p>
+
+<!-- Commenting out as redundant.
+      <p>
+        The limit on the number of concurrent queries is a <q>soft</q> one, To achieve high throughput, Impala
+        makes quick decisions at the host level about which queued queries to dispatch. Therefore, Impala might
+        slightly exceed the limits from time to time.
+      </p>
+-->
+
+      <p>
+        To avoid a large backlog of queued requests, you can set an upper limit on the size of the queue for
+        queries that are queued. When the number of queued queries exceeds this limit, further queries are
+        cancelled rather than being queued. You can also configure a timeout period per pool, after which queued queries are
+        cancelled, to avoid indefinite waits. If a cluster reaches this state where queries are cancelled due to
+        too many concurrent requests or long waits for query execution to begin, that is a signal for an
+        administrator to take action, either by provisioning more resources, scheduling work on the cluster to
+        smooth out the load, or by doing <xref href="impala_performance.xml#performance">Impala performance
+        tuning</xref> to enable higher throughput.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="admission_jdbc_odbc">
+
+    <title>How Admission Control works with Impala Clients (JDBC, ODBC, HiveServer2)</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="JDBC"/>
+      <data name="Category" value="ODBC"/>
+      <data name="Category" value="HiveServer2"/>
+      <data name="Category" value="Concepts"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        Most aspects of admission control work transparently with client interfaces such as JDBC and ODBC:
+      </p>
+
+      <ul>
+        <li>
+          If a SQL statement is put into a queue rather than running immediately, the API call blocks until the
+          statement is dequeued and begins execution. At that point, the client program can request to fetch
+          results, which might also block until results become available.
+        </li>
+
+        <li>
+          If a SQL statement is cancelled because it has been queued for too long or because it exceeded the memory
+          limit during execution, the error is returned to the client program with a descriptive error message.
+        </li>
+
+      </ul>
+
+      <p rev="CDH-27667">
+        In Impala 2.0 and higher, you can submit
+        a SQL <codeph>SET</codeph> statement from the client application
+        to change the <codeph>REQUEST_POOL</codeph> query option.
+        This option lets you submit queries to different resource pools,
+        as described in <xref href="impala_request_pool.xml#request_pool"/>.
+<!-- Commenting out as starting to be too old to mention.
+        Prior to Impala 2.0, that option was only settable
+        for a session through the <cmdname>impala-shell</cmdname> <codeph>SET</codeph> command, or cluster-wide through an
+        <cmdname>impalad</cmdname> startup option.
+-->
+      </p>
+
+      <p>
+        At any time, the set of queued queries could include queries submitted through multiple different Impala
+        daemon hosts. All the queries submitted through a particular host will be executed in order, so a
+        <codeph>CREATE TABLE</codeph> followed by an <codeph>INSERT</codeph> on the same table would succeed.
+        Queries submitted through different hosts are not guaranteed to be executed in the order they were
+        received. Therefore, if you are using load-balancing or other round-robin scheduling where different
+        statements are submitted through different hosts, set up all table structures ahead of time so that the
+        statements controlled by the queuing system are primarily queries, where order is not significant. Or, if a
+        sequence of statements needs to happen in strict order (such as an <codeph>INSERT</codeph> followed by a
+        <codeph>SELECT</codeph>), submit all those statements through a single session, while connected to the same
+        Impala daemon host.
+      </p>
+
+      <p>
+        Admission control has the following limitations or special behavior when used with JDBC or ODBC
+        applications:
+      </p>
+
+      <ul>
+        <li>
+          The other resource-related query options,
+          <codeph>RESERVATION_REQUEST_TIMEOUT</codeph> and <codeph>V_CPU_CORES</codeph>, are no longer used. Those query options only
+          applied to using Impala with Llama, which is no longer supported.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="admission_schema_config">
+    <title>SQL and Schema Considerations for Admission Control</title>
+    <conbody>
+      <p>
+        When queries complete quickly and are tuned for optimal memory usage, there is less chance of
+        performance or capacity problems during times of heavy load. Before setting up admission control,
+        tune your Impala queries to ensure that the query plans are efficient and the memory estimates
+        are accurate. Understanding the nature of your workload, and which queries are the most
+        resource-intensive, helps you to plan how to divide the queries into different pools and
+        decide what limits to define for each pool.
+      </p>
+      <p>
+        For large tables, especially those involved in join queries, keep their statistics up to date
+        after loading substantial amounts of new data or adding new partitions.
+        Use the <codeph>COMPUTE STATS</codeph> statement for unpartitioned tables, and
+        <codeph>COMPUTE INCREMENTAL STATS</codeph> for partitioned tables.
+      </p>
+      <p>
+        When you use dynamic resource pools with a <uicontrol>Max Memory</uicontrol> setting enabled,
+        you typically override the memory estimates that Impala makes based on the statistics from the
+        <codeph>COMPUTE STATS</codeph> statement.
+        You either set the <codeph>MEM_LIMIT</codeph> query option within a particular session to
+        set an upper memory limit for queries within that session, or a default <codeph>MEM_LIMIT</codeph>
+        setting for all queries processed by the <cmdname>impalad</cmdname> instance, or
+        a default <codeph>MEM_LIMIT</codeph> setting for all queries assigned to a particular
+        dynamic resource pool. By designating a consistent memory limit for a set of similar queries
+        that use the same resource pool, you avoid unnecessary query queuing or out-of-memory conditions
+        that can arise during high-concurrency workloads when memory estimates for some queries are inaccurate.
+      </p>
+      <p>
+        Follow other steps from <xref href="impala_performance.xml#performance"/> to tune your queries.
+      </p>
+    </conbody>
+  </concept>
+
+
+  <concept id="admission_config">
+
+    <title>Configuring Admission Control</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Configuring"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        The configuration options for admission control range from the simple (a single resource pool with a single
+        set of options) to the complex (multiple resource pools with different options, each pool handling queries
+        for a different set of users and groups). <ph rev="upstream">Cloudera</ph> recommends configuring the settings through the Cloudera Manager user
+        interface.
+        <!--
+        , or on a system without Cloudera Manager by editing configuration files or through startup
+        options to the <cmdname>impalad</cmdname> daemon.
+        -->
+      </p>
+
+<!-- To do: reconcile the similar notes in impala_admission.xml and admin_impala_admission_control.xml
+     and make into a conref in both places. -->
+      <note type="important">
+        Although the following options are still present in the Cloudera Manager interface under the
+        <uicontrol>Admission Control</uicontrol> configuration settings dialog,
+        <ph rev="upstream">Cloudera</ph> recommends you not use them in <keyword keyref="impala25_full"/> and higher.
+        These settings only apply if you enable admission control but leave dynamic resource pools disabled.
+        In <keyword keyref="impala25_full"/> and higher, prefer to set up dynamic resource pools and
+        customize the settings for each pool, as described in
+        <ph audience="integrated"><xref href="cm_mc_resource_pools.xml#concept_xkk_l1d_wr/section_p15_mhn_2v"/> and <xref href="cm_mc_resource_pools.xml#concept_xkk_l1d_wr/section_gph_tnk_lm"/></ph>
+        <xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_resource_pools.html" scope="external" format="html"/>.
+      </note>
+
+      <section id="admission_flags">
+
+        <title>Impala Service Flags for Admission Control (Advanced)</title>
+
+        <p>
+          The following Impala configuration options let you adjust the settings of the admission control feature. When supplying the
+          options on the <cmdname>impalad</cmdname> command line, prepend the option name with <codeph>--</codeph>.
+        </p>
+
+        <dl id="admission_control_option_list">
+          <dlentry id="queue_wait_timeout_ms">
+            <dt>
+              <codeph>queue_wait_timeout_ms</codeph>
+            </dt>
+            <dd>
+              <indexterm audience="Cloudera">--queue_wait_timeout_ms</indexterm>
+              <b>Purpose:</b> Maximum amount of time (in milliseconds) that a
+              request waits to be admitted before timing out.
+              <p>
+                <b>Type:</b> <codeph>int64</codeph>
+              </p>
+              <p>
+                <b>Default:</b> <codeph>60000</codeph>
+              </p>
+            </dd>
+          </dlentry>
+          <dlentry id="default_pool_max_requests">
+            <dt>
+              <codeph>default_pool_max_requests</codeph>
+            </dt>
+            <dd>
+              <indexterm audience="Cloudera">--default_pool_max_requests</indexterm>
+              <b>Purpose:</b> Maximum number of concurrent outstanding requests
+              allowed to run before incoming requests are queued. Because this
+              limit applies cluster-wide, but each Impala node makes independent
+              decisions to run queries immediately or queue them, it is a soft
+              limit; the overall number of concurrent queries might be slightly
+              higher during times of heavy load. A negative value indicates no
+              limit. Ignored if <codeph>fair_scheduler_config_path</codeph> and
+                <codeph>llama_site_path</codeph> are set. <p>
+                <b>Type:</b>
+                <codeph>int64</codeph>
+              </p>
+              <p>
+                <b>Default:</b>
+                <ph rev="2.5.0">-1, meaning unlimited (prior to <keyword keyref="impala25_full"/> the default was 200)</ph>
+              </p>
+            </dd>
+          </dlentry>
+          <dlentry id="default_pool_max_queued">
+            <dt>
+              <codeph>default_pool_max_queued</codeph>
+            </dt>
+            <dd>
+              <indexterm audience="Cloudera">--default_pool_max_queued</indexterm>
+              <b>Purpose:</b> Maximum number of requests allowed to be queued
+              before rejecting requests. Because this limit applies
+              cluster-wide, but each Impala node makes independent decisions to
+              run queries immediately or queue them, it is a soft limit; the
+              overall number of queued queries might be slightly higher during
+              times of heavy load. A negative value or 0 indicates requests are
+              always rejected once the maximum concurrent requests are
+              executing. Ignored if <codeph>fair_scheduler_config_path</codeph>
+              and <codeph>llama_site_path</codeph> are set. <p>
+                <b>Type:</b>
+                <codeph>int64</codeph>
+              </p>
+              <p>
+                <b>Default:</b>
+                <ph rev="2.5.0">unlimited</ph>
+              </p>
+            </dd>
+          </dlentry>
+          <dlentry id="default_pool_mem_limit">
+            <dt>
+              <codeph>default_pool_mem_limit</codeph>
+            </dt>
+            <dd>
+              <indexterm audience="Cloudera">--default_pool_mem_limit</indexterm>
+              <b>Purpose:</b> Maximum amount of memory (across the entire
+              cluster) that all outstanding requests in this pool can use before
+              new requests to this pool are queued. Specified in bytes,
+              megabytes, or gigabytes by a number followed by the suffix
+                <codeph>b</codeph> (optional), <codeph>m</codeph>, or
+                <codeph>g</codeph>, either uppercase or lowercase. You can
+              specify floating-point values for megabytes and gigabytes, to
+              represent fractional numbers such as <codeph>1.5</codeph>. You can
+              also specify it as a percentage of the physical memory by
+              specifying the suffix <codeph>%</codeph>. 0 or no setting
+              indicates no limit. Defaults to bytes if no unit is given. Because
+              this limit applies cluster-wide, but each Impala node makes
+              independent decisions to run queries immediately or queue them, it
+              is a soft limit; the overall memory used by concurrent queries
+              might be slightly higher during times of heavy load. Ignored if
+                <codeph>fair_scheduler_config_path</codeph> and
+                <codeph>llama_site_path</codeph> are set. <note
+                conref="../shared/impala_common.xml#common/admission_compute_stats" />
+              <p conref="../shared/impala_common.xml#common/type_string" />
+              <p>
+                <b>Default:</b>
+                <codeph>""</codeph> (empty string, meaning unlimited) </p>
+            </dd>
+          </dlentry>
+          <!-- Possibly from here on down, command-line controls not applicable to CM. -->
+          <dlentry id="disable_admission_control">
+            <dt>
+              <codeph>disable_admission_control</codeph>
+            </dt>
+            <dd>
+              <indexterm audience="Cloudera">--disable_admission_control</indexterm>
+              <b>Purpose:</b> Turns off the admission control feature entirely,
+              regardless of other configuration option settings.
+              <p>
+                <b>Type:</b> Boolean </p>
+              <p>
+                <b>Default:</b>
+                <codeph>false</codeph>
+              </p>
+            </dd>
+          </dlentry>
+          <dlentry id="disable_pool_max_requests">
+            <dt>
+              <codeph>disable_pool_max_requests</codeph>
+            </dt>
+            <dd>
+              <indexterm audience="Cloudera">--disable_pool_max_requests</indexterm>
+              <b>Purpose:</b> Disables all per-pool limits on the maximum number
+              of running requests. <p>
+                <b>Type:</b> Boolean </p>
+              <p>
+                <b>Default:</b>
+                <codeph>false</codeph>
+              </p>
+            </dd>
+          </dlentry>
+          <dlentry id="disable_pool_mem_limits">
+            <dt>
+              <codeph>disable_pool_mem_limits</codeph>
+            </dt>
+            <dd>
+              <indexterm audience="Cloudera">--disable_pool_mem_limits</indexterm>
+              <b>Purpose:</b> Disables all per-pool mem limits. <p>
+                <b>Type:</b> Boolean </p>
+              <p>
+                <b>Default:</b>
+                <codeph>false</codeph>
+              </p>
+            </dd>
+          </dlentry>
+          <dlentry id="fair_scheduler_allocation_path">
+            <dt>
+              <codeph>fair_scheduler_allocation_path</codeph>
+            </dt>
+            <dd>
+              <indexterm audience="Cloudera">--fair_scheduler_allocation_path</indexterm>
+              <b>Purpose:</b> Path to the fair scheduler allocation file
+                (<codeph>fair-scheduler.xml</codeph>). <p
+                conref="../shared/impala_common.xml#common/type_string" />
+              <p>
+                <b>Default:</b>
+                <codeph>""</codeph> (empty string) </p>
+              <p>
+                <b>Usage notes:</b> Admission control only uses a small subset
+                of the settings that can go in this file, as described below.
+                For details about all the Fair Scheduler configuration settings,
+                see the <xref
+                  href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html#Configuration"
+                  scope="external" format="html">Apache wiki</xref>. </p>
+            </dd>
+          </dlentry>
+          <dlentry id="llama_site_path">
+            <dt>
+              <codeph>llama_site_path</codeph>
+            </dt>
+            <dd>
+              <indexterm audience="Cloudera">--llama_site_path</indexterm>
+              <b>Purpose:</b> Path to the configuration file used by admission control
+                (<codeph>llama-site.xml</codeph>). If set,
+                <codeph>fair_scheduler_allocation_path</codeph> must also be set.
+              <p conref="../shared/impala_common.xml#common/type_string" />
+              <p>
+                <b>Default:</b> <codeph>""</codeph> (empty string) </p>
+              <p>
+                <b>Usage notes:</b> Admission control only uses a few
+                of the settings that can go in this file, as described below.
+              </p>
+            </dd>
+          </dlentry>
+        </dl>
+      </section>
+    </conbody>
+
+    <concept id="admission_config_cm">
+
+<!-- TK: Maybe all this stuff overlaps with admin_impala_admission_control and can be delegated there. -->
+
+      <title>Configuring Admission Control Using Cloudera Manager</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Cloudera Manager"/>
+    </metadata>
+  </prolog>
+
+      <conbody>
+
+        <p>
+          In Cloudera Manager, you can configure pools to manage queued Impala queries, and the options for the
+          limit on number of concurrent queries and how to handle queries that exceed the limit. For details, see
+          <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_managing_resources.html" scope="external" format="html">Managing Resources with Cloudera Manager</xref>.
+        </p>
+
+        <p audience="Cloudera"><!-- Hiding link because that subtopic is now hidden. -->
+          See <xref href="#admission_examples"/> for a sample setup for admission control under
+          Cloudera Manager.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept id="admission_config_noncm">
+
+      <title>Configuring Admission Control Using the Command Line</title>
+
+      <conbody>
+
+        <p>
+          If you do not use Cloudera Manager, you use a combination of startup options for the Impala daemon, and
+          optionally editing or manually constructing the configuration files
+          <filepath>fair-scheduler.xml</filepath> and <filepath>llama-site.xml</filepath>.
+        </p>
+
+        <p>
+          For a straightforward configuration using a single resource pool named <codeph>default</codeph>, you can
+          specify configuration options on the command line and skip the <filepath>fair-scheduler.xml</filepath>
+          and <filepath>llama-site.xml</filepath> configuration files.
+        </p>
+
+        <p>
+          For an advanced configuration with multiple resource pools using different settings, set up the
+          <filepath>fair-scheduler.xml</filepath> and <filepath>llama-site.xml</filepath> configuration files
+          manually. Provide the paths to each one using the <cmdname>impalad</cmdname> command-line options,
+          <codeph>--fair_scheduler_allocation_path</codeph> and <codeph>--llama_site_path</codeph> respectively.
+        </p>
+
+        <p>
+          The Impala admission control feature only uses the Fair Scheduler configuration settings to determine how
+          to map users and groups to different resource pools. For example, you might set up different resource
+          pools with separate memory limits, and maximum number of concurrent and queued queries, for different
+          categories of users within your organization. For details about all the Fair Scheduler configuration
+          settings, see the
+          <xref href="http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html#Configuration" scope="external" format="html">Apache
+          wiki</xref>.
+        </p>
+
+        <p>
+          The Impala admission control feature only uses a small subset of possible settings from the
+          <filepath>llama-site.xml</filepath> configuration file:
+        </p>
+
+<codeblock>llama.am.throttling.maximum.placed.reservations.<varname>queue_name</varname>
+llama.am.throttling.maximum.queued.reservations.<varname>queue_name</varname>
+<ph rev="2.5.0 IMPALA-2538">impala.admission-control.pool-default-query-options.<varname>queue_name</varname>
+impala.admission-control.pool-queue-timeout-ms.<varname>queue_name</varname></ph>
+</codeblock>
+
+        <p rev="2.5.0 IMPALA-2538">
+          The <codeph>impala.admission-control.pool-queue-timeout-ms</codeph>
+          setting specifies the timeout value for this pool, in milliseconds.
+          The<codeph>impala.admission-control.pool-default-query-options</codeph>
+          settings designates the default query options for all queries that run
+          in this pool. Its argument value is a comma-delimited string of
+          'key=value' pairs, for example,<codeph>'key1=val1,key2=val2'</codeph>.
+          For example, this is where you might set a default memory limit
+          for all queries in the pool, using an argument such as <codeph>MEM_LIMIT=5G</codeph>.
+        </p>
+
+        <p rev="2.5.0 IMPALA-2538">
+          The <codeph>impala.admission-control.*</codeph> configuration settings are available in
+          <keyword keyref="impala25_full"/> and higher.
+        </p>
+
+        <p audience="Cloudera"><!-- Hiding link because that subtopic is now hidden. -->
+          See <xref href="#admission_examples/section_etq_qgb_rq"/> for sample configuration files
+          for admission control using multiple resource pools, without Cloudera Manager.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept id="admission_examples">
+    <!-- Pruning the CM examples and screenshots because in Impala 2.5 the defaults match up much better with our recommendations. -->
+
+      <title>Examples of Admission Control Configurations</title>
+
+      <conbody>
+
+        <section id="section_fqn_qgb_rq">
+
+          <title>Example Admission Control Configurations Using Cloudera Manager</title>
+
+          <p>
+            For full instructions about configuring dynamic resource pools through Cloudera Manager, see
+            <xref audience="integrated" href="cm_mc_resource_pools.xml#xd_583c10bfdbd326ba--43d5fd93-1410993f8c2--7ff2"/><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_resource_pools.html" scope="external" format="html"/>.
+          </p>
+
+        </section>
+
+        <section id="section_etq_qgb_rq">
+
+          <title>Example Admission Control Configurations Using Configuration Files</title>
+
+          <p>
+            For clusters not managed by Cloudera Manager, here are sample <filepath>fair-scheduler.xml</filepath>
+            and <filepath>llama-site.xml</filepath> files that define resource pools <codeph>root.default</codeph>,
+            <codeph>root.development</codeph>, and <codeph>root.production</codeph>.
+            These sample files are stripped down: in a real deployment they
+            might contain other settings for use with various aspects of the YARN component. The
+            settings shown here are the significant ones for the Impala admission control feature.
+          </p>
+
+          <p>
+            <b>fair-scheduler.xml:</b>
+          </p>
+
+          <p>
+            Although Impala does not use the <codeph>vcores</codeph> value, you must still specify it to satisfy
+            YARN requirements for the file contents.
+          </p>
+
+          <p>
+            Each <codeph>&lt;aclSubmitApps&gt;</codeph> tag (other than the one for <codeph>root</codeph>) contains
+            a comma-separated list of users, then a space, then a comma-separated list of groups; these are the
+            users and groups allowed to submit Impala statements to the corresponding resource pool.
+          </p>
+
+          <p>
+            If you leave the <codeph>&lt;aclSubmitApps&gt;</codeph> element empty for a pool, nobody can submit
+            directly to that pool; child pools can specify their own <codeph>&lt;aclSubmitApps&gt;</codeph> values
+            to authorize users and groups to submit to those pools.
+          </p>
+
+<codeblock><![CDATA[<allocations>
+    <queue name="root">
+        <aclSubmitApps> </aclSubmitApps>
+        <queue name="default">
+            <maxResources>50000 mb, 0 vcores</maxResources>
+            <aclSubmitApps>*</aclSubmitApps>
+        </queue>
+        <queue name="development">
+            <maxResources>200000 mb, 0 vcores</maxResources>
+            <aclSubmitApps>user1,user2 dev,ops,admin</aclSubmitApps>
+        </queue>
+        <queue name="production">
+            <maxResources>1000000 mb, 0 vcores</maxResources>
+            <aclSubmitApps> ops,admin</aclSubmitApps>
+        </queue>
+    </queue>
+    <queuePlacementPolicy>
+        <rule name="specified" create="false"/>
+        <rule name="default" />
+    </queuePlacementPolicy>
+</allocations>
+]]>
+</codeblock>
+
+          <p>
+            <b>llama-site.xml:</b>
+          </p>
+
+<codeblock rev="2.5.0 IMPALA-2538"><![CDATA[
+<?xml version="1.0" encoding="UTF-8"?>
+<configuration>
+  <property>
+    <name>llama.am.throttling.maximum.placed.reservations.root.default</name>
+    <value>10</value>
+  </property>
+  <property>
+    <name>llama.am.throttling.maximum.queued.reservations.root.default</name>
+    <value>50</value>
+  </property>
+  <property>
+    <name>impala.admission-control.pool-default-query-options.root.default</name>
+    <value>mem_limit=128m,query_timeout_s=20,max_io_buffers=10</value>
+  </property>
+  <property>
+    <name>impala.admission-control.pool-queue-timeout-ms.root.default</name>
+    <value>30000</value>
+  </property>
+  <property>
+    <name>llama.am.throttling.maximum.placed.reservations.root.development</name>
+    <value>50</value>
+  </property>
+  <property>
+    <name>llama.am.throttling.maximum.queued.reservations.root.development</name>
+    <value>100</value>
+  </property>
+  <property>
+    <name>impala.admission-control.pool-default-query-options.root.development</name>
+    <value>mem_limit=256m,query_timeout_s=30,max_io_buffers=10</value>
+  </property>
+  <property>
+    <name>impala.admission-control.pool-queue-timeout-ms.root.development</name>
+    <value>15000</value>
+  </property>
+  <property>
+    <name>llama.am.throttling.maximum.placed.reservations.root.production</name>
+    <value>100</value>
+  </property>
+  <property>
+    <name>llama.am.throttling.maximum.queued.reservations.root.production</name>
+    <value>200</value>
+  </property>
+<!--
+       Default query options for the 'root.production' pool.
+       THIS IS A NEW PARAMETER in CDH 5.7 / Impala 2.5.
+       Note that the MEM_LIMIT query option still shows up in here even though it is a
+       separate box in the UI. We do that because it is the most important query option
+       that people will need (everything else is somewhat advanced).
+
+       MEM_LIMIT takes a per-node memory limit which is specified using one of the following:
+        - '<int>[bB]?'  -> bytes (default if no unit given)
+        - '<float>[mM(bB)]' -> megabytes
+        - '<float>[gG(bB)]' -> in gigabytes
+        E.g. 'MEM_LIMIT=12345' (no unit) means 12345 bytes, and you can append m or g
+             to specify megabytes or gigabytes, though that is not required.
+-->
+  <property>
+    <name>impala.admission-control.pool-default-query-options.root.production</name>
+    <value>mem_limit=386m,query_timeout_s=30,max_io_buffers=10</value>
+  </property>
+<!--
+  Default queue timeout (ms) for the pool 'root.production'.
+  If this isn\u2019t set, the process-wide flag is used.
+  THIS IS A NEW PARAMETER in CDH 5.7 / Impala 2.5.
+-->
+  <property>
+    <name>impala.admission-control.pool-queue-timeout-ms.root.production</name>
+    <value>30000</value>
+  </property>
+</configuration>
+]]>
+</codeblock>
+        </section>
+      </conbody>
+    </concept>
+  </concept>
+
+<!-- End Config -->
+
+  <concept id="admission_guidelines">
+
+    <title>Guidelines for Using Admission Control</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Planning"/>
+      <data name="Category" value="Guidelines"/>
+      <data name="Category" value="Best Practices"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        To see how admission control works for particular queries, examine the profile output for the query. This
+        information is available through the <codeph>PROFILE</codeph> statement in <cmdname>impala-shell</cmdname>
+        immediately after running a query in the shell, on the <uicontrol>queries</uicontrol> page of the Impala
+        debug web UI, or in the Impala log file (basic information at log level 1, more detailed information at log
+        level 2). The profile output contains details about the admission decision, such as whether the query was
+        queued or not and which resource pool it was assigned to. It also includes the estimated and actual memory
+        usage for the query, so you can fine-tune the configuration for the memory limits of the resource pools.
+      </p>
+
+      <p>
+        Where practical, use Cloudera Manager to configure the admission control parameters. The Cloudera Manager
+        GUI is much simpler than editing the configuration files directly.
+      </p>
+
+      <p>
+        Remember that the limits imposed by admission control are <q>soft</q> limits.
+        The decentralized nature of this mechanism means that each Impala node makes its own decisions about whether
+        to allow queries to run immediately or to queue them. These decisions rely on information passed back and forth
+        between nodes by the statestore service. If a sudden surge in requests causes more queries than anticipated to run
+        concurrently, then throughput could decrease due to queries spilling to disk or contending for resources;
+        or queries could be cancelled if they exceed the <codeph>MEM_LIMIT</codeph> setting while running.
+      </p>
+
+<!--
+      <p>
+        If you have trouble getting a query to run because its estimated memory usage is too high, you can override
+        the estimate by setting the <codeph>MEM_LIMIT</codeph> query option in <cmdname>impala-shell</cmdname>,
+        then issuing the query through the shell in the same session. The <codeph>MEM_LIMIT</codeph> value is
+        treated as the estimated amount of memory, overriding the estimate that Impala would generate based on
+        table and column statistics. This value is used only for making admission control decisions, and is not
+        pre-allocated by the query.
+      </p>
+-->
+
+      <p>
+        In <cmdname>impala-shell</cmdname>, you can also specify which resource pool to direct queries to by
+        setting the <codeph>REQUEST_POOL</codeph> query option.
+      </p>
+
+      <p>
+        The statements affected by the admission control feature are primarily queries, but also include statements
+        that write data such as <codeph>INSERT</codeph> and <codeph>CREATE TABLE AS SELECT</codeph>. Most write
+        operations in Impala are not resource-intensive, but inserting into a Parquet table can require substantial
+        memory due to buffering intermediate data before writing out each Parquet data block. See
+        <xref href="impala_parquet.xml#parquet_etl"/> for instructions about inserting data efficiently into
+        Parquet tables.
+      </p>
+
+      <p>
+        Although admission control does not scrutinize memory usage for other kinds of DDL statements, if a query
+        is queued due to a limit on concurrent queries or memory usage, subsequent statements in the same session
+        are also queued so that they are processed in the correct order:
+      </p>
+
+<codeblock>-- This query could be queued to avoid out-of-memory at times of heavy load.
+select * from huge_table join enormous_table using (id);
+-- If so, this subsequent statement in the same session is also queued
+-- until the previous statement completes.
+drop table huge_table;
+</codeblock>
+
+      <p>
+        If you set up different resource pools for different users and groups, consider reusing any classifications
+        you developed for use with Sentry security. See <xref href="impala_authorization.xml#authorization"/> for details.
+      </p>
+
+      <p>
+        For details about all the Fair Scheduler configuration settings, see
+        <xref href="https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-site/FairScheduler.html#Configuration" scope="external" format="html">Fair
+        Scheduler Configuration</xref>, in particular the tags such as <codeph>&lt;queue&gt;</codeph> and
+        <codeph>&lt;aclSubmitApps&gt;</codeph> to map users and groups to particular resource pools (queues).
+      </p>
+
+<!-- Wait a sec. We say admission control doesn't use RESERVATION_REQUEST_TIMEOUT at all.
+     What's the real story here? Matt did refer to some timeout option that was
+     available through the shell but not the DB-centric APIs.
+<p>
+  Because you cannot override query options such as
+  <codeph>RESERVATION_REQUEST_TIMEOUT</codeph>
+  in a JDBC or ODBC application, consider configuring timeout periods
+  on the application side to cancel queries that take
+  too long due to being queued during times of high load.
+</p>
+-->
+    </conbody>
+  </concept>
+</concept>
+<!-- Admission control -->

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_aggregate_functions.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_aggregate_functions.xml b/docs/topics/impala_aggregate_functions.xml
new file mode 100644
index 0000000..66d617a
--- /dev/null
+++ b/docs/topics/impala_aggregate_functions.xml
@@ -0,0 +1,33 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="aggregate_functions">
+
+  <title>Impala Aggregate Functions</title>
+  <titlealts audience="PDF"><navtitle>Aggregate Functions</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="Aggregate Functions"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Querying"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p conref="../shared/impala_common.xml#common/aggr1"/>
+
+<codeblock conref="../shared/impala_common.xml#common/aggr2"/>
+
+    <p conref="../shared/impala_common.xml#common/aggr3"/>
+
+    <p>
+      <indexterm audience="Cloudera">aggregate functions</indexterm>
+    </p>
+
+    <p outputclass="toc"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_aliases.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_aliases.xml b/docs/topics/impala_aliases.xml
new file mode 100644
index 0000000..fd686cd
--- /dev/null
+++ b/docs/topics/impala_aliases.xml
@@ -0,0 +1,87 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="aliases">
+
+  <title>Overview of Impala Aliases</title>
+  <titlealts audience="PDF"><navtitle>Aliases</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      When you write the names of tables, columns, or column expressions in a query, you can assign an alias at the
+      same time. Then you can specify the alias rather than the original name when making other references to the
+      table or column in the same statement. You typically specify aliases that are shorter, easier to remember, or
+      both than the original names. The aliases are printed in the query header, making them useful for
+      self-documenting output.
+    </p>
+
+    <p>
+      To set up an alias, add the <codeph>AS <varname>alias</varname></codeph> clause immediately after any table,
+      column, or expression name in the <codeph>SELECT</codeph> list or <codeph>FROM</codeph> list of a query. The
+      <codeph>AS</codeph> keyword is optional; you can also specify the alias immediately after the original name.
+    </p>
+
+<codeblock>-- Make the column headers of the result set easier to understand.
+SELECT c1 AS name, c2 AS address, c3 AS phone FROM table_with_terse_columns;
+SELECT SUM(ss_xyz_dollars_net) AS total_sales FROM table_with_cryptic_columns;
+-- The alias can be a quoted string for extra readability.
+SELECT c1 AS "Employee ID", c2 AS "Date of hire" FROM t1;
+-- The AS keyword is optional.
+SELECT c1 "Employee ID", c2 "Date of hire" FROM t1;
+
+-- The table aliases assigned in the FROM clause can be used both earlier
+-- in the query (the SELECT list) and later (the WHERE clause).
+SELECT one.name, two.address, three.phone
+  FROM census one, building_directory two, phonebook three
+WHERE one.id = two.id and two.id = three.id;
+
+-- The aliases c1 and c2 let the query handle columns with the same names from 2 joined tables.
+-- The aliases t1 and t2 let the query abbreviate references to long or cryptically named tables.
+SELECT t1.column_n AS c1, t2.column_n AS c2 FROM long_name_table AS t1, very_long_name_table2 AS t2
+  WHERE c1 = c2;
+SELECT t1.column_n c1, t2.column_n c2 FROM table1 t1, table2 t2
+  WHERE c1 = c2;
+</codeblock>
+
+    <p>
+      To use an alias name that matches one of the Impala reserved keywords (listed in
+      <xref href="impala_reserved_words.xml#reserved_words"/>), surround the identifier with either single or
+      double quotation marks, or <codeph>``</codeph> characters (backticks).
+    </p>
+
+    <p>
+      <ph conref="../shared/impala_common.xml#common/aliases_vs_identifiers"/>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+    <p rev="2.3.0">
+      Queries involving the complex types (<codeph>ARRAY</codeph>,
+      <codeph>STRUCT</codeph>, and <codeph>MAP</codeph>), typically make
+      extensive use of table aliases. These queries involve join clauses
+      where the complex type column is treated as a joined table.
+      To construct two-part or three-part qualified names for the
+      complex column elements in the <codeph>FROM</codeph> list,
+      sometimes it is syntactically required to construct a table
+      alias for the complex column where it is referenced in the join clause.
+      See <xref href="impala_complex_types.xml#complex_types"/> for details and examples.
+    </p>
+
+    <p>
+      <b>Alternatives:</b>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/views_vs_identifiers"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_allow_unsupported_formats.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_allow_unsupported_formats.xml b/docs/topics/impala_allow_unsupported_formats.xml
new file mode 100644
index 0000000..fbf758e
--- /dev/null
+++ b/docs/topics/impala_allow_unsupported_formats.xml
@@ -0,0 +1,31 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="allow_unsupported_formats">
+
+  <title>ALLOW_UNSUPPORTED_FORMATS Query Option</title>
+  <titlealts audience="PDF"><navtitle>ALLOW_UNSUPPORTED_FORMATS</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Deprecated Features"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+<!--
+The original brief explanation with not enough detail comes from the comments at:
+  http://github.sf.cloudera.com/CDH/Impala/raw/master/common/thrift/ImpalaService.thrift
+Removing that wording from here after discussions with dev team. Just recording the URL for posterity.
+-->
+
+    <p>
+      An obsolete query option from early work on support for file formats. Do not use. Might be removed in the
+      future.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_boolean"/>
+    <p conref="../shared/impala_common.xml#common/default_false_0"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_alter_function.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_alter_function.xml b/docs/topics/impala_alter_function.xml
new file mode 100644
index 0000000..2ea0a26
--- /dev/null
+++ b/docs/topics/impala_alter_function.xml
@@ -0,0 +1,21 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept audience="Cloudera" rev="1.x" id="alter_function">
+
+  <title>ALTER FUNCTION Statement</title>
+  <titlealts audience="PDF"><navtitle>ALTER FUNCTION</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p/>
+  </conbody>
+</concept>


[09/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_s3.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_s3.xml b/docs/topics/impala_s3.xml
new file mode 100644
index 0000000..e61bb7c
--- /dev/null
+++ b/docs/topics/impala_s3.xml
@@ -0,0 +1,796 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="s3" rev="2.2.0">
+
+  <title>Using Impala with the Amazon S3 Filesystem</title>
+  <titlealts audience="PDF"><navtitle>S3 Tables</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Amazon"/>
+      <data name="Category" value="S3"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Preview Features"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <note conref="../shared/impala_common.xml#common/s3_production"/>
+
+    <p rev="2.2.0">
+      <indexterm audience="Cloudera">S3 with Impala</indexterm>
+
+      <indexterm audience="Cloudera">Amazon S3 with Impala</indexterm>
+      You can use Impala to query data residing on the Amazon S3 filesystem. This capability allows convenient
+      access to a storage system that is remotely managed, accessible from anywhere, and integrated with various
+      cloud-based services. Impala can query files in any supported file format from S3. The S3 storage location
+      can be for an entire table, or individual partitions in a partitioned table.
+    </p>
+
+    <p>
+      The default Impala tables use data files stored on HDFS, which are ideal for bulk loads and queries using
+      full-table scans. In contrast, queries against S3 data are less performant, making S3 suitable for holding
+      <q>cold</q> data that is only queried occasionally, while more frequently accessed <q>hot</q> data resides in
+      HDFS. In a partitioned table, you can set the <codeph>LOCATION</codeph> attribute for individual partitions
+      to put some partitions on HDFS and others on S3, typically depending on the age of the data.
+    </p>
+
+    <p outputclass="toc inpage"/>
+
+  </conbody>
+
+  <concept id="s3_sql">
+    <title>How Impala SQL Statements Work with S3</title>
+    <conbody>
+      <p>
+        Impala SQL statements work with data on S3 as follows:
+      </p>
+      <ul>
+        <li>
+          <p>
+            The <xref href="impala_create_table.xml#create_table"/>
+            or <xref href="impala_alter_table.xml#alter_table"/> statements
+            can specify that a table resides on the S3 filesystem by
+            encoding an <codeph>s3a://</codeph> prefix for the <codeph>LOCATION</codeph>
+            property. <codeph>ALTER TABLE</codeph> can also set the <codeph>LOCATION</codeph>
+            property for an individual partition, so that some data in a table resides on
+            S3 and other data in the same table resides on HDFS.
+          </p>
+        </li>
+        <li>
+          <p>
+            Once a table or partition is designated as residing on S3, the <xref href="impala_select.xml#select"/>
+            statement transparently accesses the data files from the appropriate storage layer.
+          </p>
+        </li>
+        <li>
+          <p>
+            If the S3 table is an internal table, the <xref href="impala_drop_table.xml#drop_table"/> statement
+            removes the corresponding data files from S3 when the table is dropped.
+          </p>
+        </li>
+        <li>
+          <p>
+            The <xref href="impala_truncate_table.xml#truncate_table"/> statement always removes the corresponding
+            data files from S3 when the table is truncated.
+          </p>
+        </li>
+        <li>
+          <p>
+            The <xref href="impala_load_data.xml#load_data"/> can move data files residing in HDFS into
+            an S3 table.
+          </p>
+        </li>
+        <li>
+          <p>
+            The <xref href="impala_insert.xml#insert"/> statement, or the <codeph>CREATE TABLE AS SELECT</codeph>
+            form of the <codeph>CREATE TABLE</codeph> statement, can copy data from an HDFS table or another S3
+            table into an S3 table. The <xref href="impala_s3_skip_insert_staging.xml#s3_skip_insert_staging"/>
+            query option chooses whether or not to use a fast code path for these write operations to S3,
+            with the tradeoff of potential inconsistency in the case of a failure during the statement. 
+          </p>
+        </li>
+      </ul>
+      <p>
+        For usage information about Impala SQL statements with S3 tables, see <xref href="impala_s3.xml#s3_ddl"/>
+        and <xref href="impala_s3.xml#s3_dml"/>.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="s3_creds">
+
+    <title>Specifying Impala Credentials to Access Data in S3</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">fs.s3a.access.key configuration setting</indexterm>
+        <indexterm audience="Cloudera">fs.s3a.secret.key configuration setting</indexterm>
+        <indexterm audience="Cloudera">access.key configuration setting</indexterm>
+        <indexterm audience="Cloudera">secret.key configuration setting</indexterm>
+        To allow Impala to access data in S3, specify values for the following configuration settings in your
+        <filepath>core-site.xml</filepath> file:
+      </p>
+
+<!-- Normally I would turn this example into CDATA notation to avoid all the &lt; and &gt; entities.
+     However, then I couldn't use the <varname> tag inside the same example. -->
+<codeblock>
+&lt;property&gt;
+&lt;name&gt;fs.s3a.access.key&lt;/name&gt;
+&lt;value&gt;<varname>your_access_key</varname>&lt;/value&gt;
+&lt;/property&gt;
+&lt;property&gt;
+&lt;name&gt;fs.s3a.secret.key&lt;/name&gt;
+&lt;value&gt;<varname>your_secret_key</varname>&lt;/value&gt;
+&lt;/property&gt;
+</codeblock>
+
+      <p>
+        As of CDH 5.8, these settings do not have corresponding controls in the Cloudera Manager user interface.
+        Specify them in the <uicontrol>HDFS Client Advanced Configuration Snippet (Safety Valve) for
+        core-site.xml</uicontrol> field. After specifying the credentials, restart both the Impala and Hive
+        services. (Restarting Hive is required because Impala queries, <codeph>CREATE TABLE</codeph> statements,
+        and so on go through the Hive metastore.)
+      </p>
+
+<!--
+      <p rev="CDH-39914 IMPALA-3306">
+        In <keyword keyref="impala26_full"/> and higher, you can specify the S3 access key and secret key through
+        configuration settings for the <cmdname>impalad</cmdname> daemon.
+        Rather than specifying the keys themselves on the command line or in startup scripts,
+        you specify the commands to retrieve the keys as <cmdname>impalad</cmdname>
+        startup options. For clusters not managed by Cloudera Manager, use the
+        <codeph>-
+        -s3a_access_key_cmd</codeph> and <codeph>-
+        -s3a_secret_key_cmd</codeph>
+        startup options for the <cmdname>impalad</cmdname> daemon.
+        For clusters managed by Cloudera Manager, set the
+        <codeph>s3a_access_key_cmd</codeph> and <codeph>s3a_secret_key_cmd</codeph>
+        configuration settings and restart the Impala and Hive services.
+        (Restarting Hive is required because Impala queries, <codeph>CREATE TABLE</codeph> statements,
+        and so on go through the Hive metastore.)
+      </p>
+-->
+
+      <note type="important">
+<!--
+      <ul>
+        <li>
+          <p rev="CDH-39914 IMPALA-3306">
+            The <codeph>s3a_access_key_cmd</codeph> and <codeph>s3a_secret_key_cmd</codeph> settings
+            for <cmdname>impalad</cmdname> only allow Impala to access S3. You must still include the credentials in the
+            client <filepath>hdfs-site.xml</filepath> configuration file to allow S3 access for the Hive metastore,
+            <codeph>hadoop fs</codeph> command, and so on.
+          </p>
+        </li>
+        <li>
+-->
+          <p>
+            Although you can specify the access key ID and secret key as part of the <codeph>s3a://</codeph> URL in the
+            <codeph>LOCATION</codeph> attribute, doing so makes this sensitive information visible in many places, such
+            as <codeph>DESCRIBE FORMATTED</codeph> output and Impala log files. Therefore, specify this information
+            centrally in the <filepath>core-site.xml</filepath> file, and restrict read access to that file to only
+            trusted users.
+          </p>
+<!--
+        </li>
+-->
+        <!-- Overriding with a new first list bullet following clarification by Sailesh.
+        <li>
+          <p rev="CDH-39914 IMPALA-3306">
+            Prior to <keyword keyref="impala26_full"/> an alternative way to specify the keys was by
+            including the fields <codeph>fs.s3a.access.key</codeph> and <codeph>fs.s3a.secret.key</codeph>
+            in a configuration file such as <filepath>core-site.xml</filepath> or <filepath>hdfs-site.xml</filepath>.
+            With the enhanced S3 key management in <keyword keyref="impala26_full"/> and higher, if you are upgrading from
+            an earlier release where you used Impala with S3, remove the S3 keys from any copies of those files.
+          </p>
+        </li>
+        -->
+<!--
+      </ul>
+-->
+      </note>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="s3_etl">
+
+    <title>Loading Data into S3 for Impala Queries</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="ETL"/>
+      <data name="Category" value="Ingest"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        If your ETL pipeline involves moving data into S3 and then querying through Impala,
+        you can either use Impala DML statements to create, move, or copy the data, or
+        use the same data loading techniques as you would for non-Impala data.
+      </p>
+
+    </conbody>
+
+    <concept id="s3_dml" rev="2.6.0 CDH-39913 IMPALA-1878">
+      <title>Using Impala DML Statements for S3 Data</title>
+      <conbody>
+        <p conref="../shared/impala_common.xml#common/s3_dml"/>
+        <p conref="../shared/impala_common.xml#common/s3_dml_performance"/>
+      </conbody>
+    </concept>
+
+    <concept id="s3_manual_etl">
+      <title>Manually Loading Data into Impala Tables on S3</title>
+      <conbody>
+        <p>
+          As an alternative, or on earlier Impala releases without DML support for S3,
+          you can use the Amazon-provided methods to bring data files into S3 for querying through Impala. See
+          <xref href="http://aws.amazon.com/s3/" scope="external" format="html">the Amazon S3 web site</xref> for
+          details.
+        </p>
+
+        <note type="important">
+          <p conref="../shared/impala_common.xml#common/s3_drop_table_purge"/>
+        </note>
+
+        <p>
+          Alternative file creation techniques (less compatible with the <codeph>PURGE</codeph> clause) include:
+        </p>
+
+        <ul>
+          <li>
+            The <xref href="https://console.aws.amazon.com/s3/home" scope="external" format="html">Amazon AWS / S3
+            web interface</xref> to upload from a web browser.
+          </li>
+
+          <li>
+            The <xref href="http://aws.amazon.com/cli/" scope="external" format="html">Amazon AWS CLI</xref> to
+            manipulate files from the command line.
+          </li>
+
+          <li>
+            Other S3-enabled software, such as
+            <xref href="http://s3tools.org/s3cmd" scope="external" format="html">the S3Tools client software</xref>.
+          </li>
+        </ul>
+
+        <p>
+          After you upload data files to a location already mapped to an Impala table or partition, or if you delete
+          files in S3 from such a location, issue the <codeph>REFRESH <varname>table_name</varname></codeph>
+          statement to make Impala aware of the new set of data files.
+        </p>
+
+      </conbody>
+    </concept>
+
+  </concept>
+
+  <concept id="s3_ddl">
+
+    <title>Creating Impala Databases, Tables, and Partitions for Data Stored on S3</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Databases"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        Impala reads data for a table or partition from S3 based on the <codeph>LOCATION</codeph> attribute for the
+        table or partition. Specify the S3 details in the <codeph>LOCATION</codeph> clause of a <codeph>CREATE
+        TABLE</codeph> or <codeph>ALTER TABLE</codeph> statement. The notation for the <codeph>LOCATION</codeph>
+        clause is <codeph>s3a://<varname>bucket_name</varname>/<varname>path/to/file</varname></codeph>. The
+        filesystem prefix is always <codeph>s3a://</codeph> because Impala does not support the <codeph>s3://</codeph> or
+        <codeph>s3n://</codeph> prefixes.
+      </p>
+
+      <p>
+        For a partitioned table, either specify a separate <codeph>LOCATION</codeph> clause for each new partition,
+        or specify a base <codeph>LOCATION</codeph> for the table and set up a directory structure in S3 to mirror
+        the way Impala partitioned tables are structured in HDFS. Although, strictly speaking, S3 filenames do not
+        have directory paths, Impala treats S3 filenames with <codeph>/</codeph> characters the same as HDFS
+        pathnames that include directories.
+      </p>
+
+      <p>
+        You point a nonpartitioned table or an individual partition at S3 by specifying a single directory
+        path in S3, which could be any arbitrary directory. To replicate the structure of an entire Impala
+        partitioned table or database in S3 requires more care, with directories and subdirectories nested and
+        named to match the equivalent directory tree in HDFS. Consider setting up an empty staging area if
+        necessary in HDFS, and recording the complete directory structure so that you can replicate it in S3.
+        <!--
+        Or, specify an S3 location for an entire database, after which all tables and partitions created inside that
+        database automatically inherit the database <codeph>LOCATION</codeph> and create new S3 directories
+        underneath the database directory.
+        -->
+      </p>
+
+      <p>
+        For convenience when working with multiple tables with data files stored in S3, you can create a database
+        with a <codeph>LOCATION</codeph> attribute pointing to an S3 path.
+        Specify a URL of the form <codeph>s3a://<varname>bucket</varname>/<varname>root/path/for/database</varname></codeph>
+        for the <codeph>LOCATION</codeph> attribute of the database.
+        Any tables created inside that database
+        automatically create directories underneath the one specified by the database
+        <codeph>LOCATION</codeph> attribute.
+      </p>
+
+      <p>
+        For example, the following session creates a partitioned table where only a single partition resides on S3.
+        The partitions for years 2013 and 2014 are located on HDFS. The partition for year 2015 includes a
+        <codeph>LOCATION</codeph> attribute with an <codeph>s3a://</codeph> URL, and so refers to data residing on
+        S3, under a specific path underneath the bucket <codeph>impala-demo</codeph>.
+      </p>
+
+<codeblock>[localhost:21000] > create database db_on_hdfs;
+[localhost:21000] > use db_on_hdfs;
+[localhost:21000] > create table mostly_on_hdfs (x int) partitioned by (year int);
+[localhost:21000] > alter table mostly_on_hdfs add partition (year=2013);
+[localhost:21000] > alter table mostly_on_hdfs add partition (year=2014);
+[localhost:21000] > alter table mostly_on_hdfs add partition (year=2015)
+                  >   location 's3a://impala-demo/dir1/dir2/dir3/t1';
+</codeblock>
+
+      <p>
+        The following session creates a database and two partitioned tables residing entirely on S3, one
+        partitioned by a single column and the other partitioned by multiple columns. Because a
+        <codeph>LOCATION</codeph> attribute with an <codeph>s3a://</codeph> URL is specified for the database, the
+        tables inside that database are automatically created on S3 underneath the database directory. To see the
+        names of the associated subdirectories, including the partition key values, we use an S3 client tool to
+        examine how the directory structure is organized on S3. For example, Impala partition directories such as
+        <codeph>month=1</codeph> do not include leading zeroes, which sometimes appear in partition directories created
+        through Hive.
+      </p>
+
+<codeblock>[localhost:21000] > create database db_on_s3 location 's3a://impala-demo/dir1/dir2/dir3';
+[localhost:21000] > use db_on_s3;
+
+[localhost:21000] > create table partitioned_on_s3 (x int) partitioned by (year int);
+[localhost:21000] > alter table partitioned_on_s3 add partition (year=2013);
+[localhost:21000] > alter table partitioned_on_s3 add partition (year=2014);
+[localhost:21000] > alter table partitioned_on_s3 add partition (year=2015);
+
+[localhost:21000] > !aws s3 ls s3://impala-demo/dir1/dir2/dir3 --recursive;
+2015-03-17 13:56:34          0 dir1/dir2/dir3/
+2015-03-17 16:43:28          0 dir1/dir2/dir3/partitioned_on_s3/
+2015-03-17 16:43:49          0 dir1/dir2/dir3/partitioned_on_s3/year=2013/
+2015-03-17 16:43:53          0 dir1/dir2/dir3/partitioned_on_s3/year=2014/
+2015-03-17 16:43:58          0 dir1/dir2/dir3/partitioned_on_s3/year=2015/
+
+[localhost:21000] > create table partitioned_multiple_keys (x int)
+                  >   partitioned by (year smallint, month tinyint, day tinyint);
+[localhost:21000] > alter table partitioned_multiple_keys
+                  >   add partition (year=2015,month=1,day=1);
+[localhost:21000] > alter table partitioned_multiple_keys
+                  >   add partition (year=2015,month=1,day=31);
+[localhost:21000] > alter table partitioned_multiple_keys
+                  >   add partition (year=2015,month=2,day=28);
+
+[localhost:21000] > !aws s3 ls s3://impala-demo/dir1/dir2/dir3 --recursive;
+2015-03-17 13:56:34          0 dir1/dir2/dir3/
+2015-03-17 16:47:13          0 dir1/dir2/dir3/partitioned_multiple_keys/
+2015-03-17 16:47:44          0 dir1/dir2/dir3/partitioned_multiple_keys/year=2015/month=1/day=1/
+2015-03-17 16:47:50          0 dir1/dir2/dir3/partitioned_multiple_keys/year=2015/month=1/day=31/
+2015-03-17 16:47:57          0 dir1/dir2/dir3/partitioned_multiple_keys/year=2015/month=2/day=28/
+2015-03-17 16:43:28          0 dir1/dir2/dir3/partitioned_on_s3/
+2015-03-17 16:43:49          0 dir1/dir2/dir3/partitioned_on_s3/year=2013/
+2015-03-17 16:43:53          0 dir1/dir2/dir3/partitioned_on_s3/year=2014/
+2015-03-17 16:43:58          0 dir1/dir2/dir3/partitioned_on_s3/year=2015/
+</codeblock>
+
+      <p>
+        The <codeph>CREATE DATABASE</codeph> and <codeph>CREATE TABLE</codeph> statements create the associated
+        directory paths if they do not already exist. You can specify multiple levels of directories, and the
+        <codeph>CREATE</codeph> statement creates all appropriate levels, similar to using <codeph>mkdir
+        -p</codeph>.
+      </p>
+
+      <p>
+        Use the standard S3 file upload methods to actually put the data files into the right locations. You can
+        also put the directory paths and data files in place before creating the associated Impala databases or
+        tables, and Impala automatically uses the data from the appropriate location after the associated databases
+        and tables are created.
+      </p>
+
+      <p>
+        You can switch whether an existing table or partition points to data in HDFS or S3. For example, if you
+        have an Impala table or partition pointing to data files in HDFS or S3, and you later transfer those data
+        files to the other filesystem, use an <codeph>ALTER TABLE</codeph> statement to adjust the
+        <codeph>LOCATION</codeph> attribute of the corresponding table or partition to reflect that change. Because
+        Impala does not have an <codeph>ALTER DATABASE</codeph> statement, this location-switching technique is not
+        practical for entire databases that have a custom <codeph>LOCATION</codeph> attribute.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="s3_internal_external">
+
+    <title>Internal and External Tables Located on S3</title>
+
+    <conbody>
+
+      <p>
+        Just as with tables located on HDFS storage, you can designate S3-based tables as either internal (managed
+        by Impala) or external, by using the syntax <codeph>CREATE TABLE</codeph> or <codeph>CREATE EXTERNAL
+        TABLE</codeph> respectively. When you drop an internal table, the files associated with the table are
+        removed, even if they are on S3 storage. When you drop an external table, the files associated with the
+        table are left alone, and are still available for access by other tools or components. See
+        <xref href="impala_tables.xml#tables"/> for details.
+      </p>
+
+      <p>
+        If the data on S3 is intended to be long-lived and accessed by other tools in addition to Impala, create
+        any associated S3 tables with the <codeph>CREATE EXTERNAL TABLE</codeph> syntax, so that the files are not
+        deleted from S3 when the table is dropped.
+      </p>
+
+      <p>
+        If the data on S3 is only needed for querying by Impala and can be safely discarded once the Impala
+        workflow is complete, create the associated S3 tables using the <codeph>CREATE TABLE</codeph> syntax, so
+        that dropping the table also deletes the corresponding data files on S3.
+      </p>
+
+      <p>
+        For example, this session creates a table in S3 with the same column layout as a table in HDFS, then
+        examines the S3 table and queries some data from it. The table in S3 works the same as a table in HDFS as
+        far as the expected file format of the data, table and column statistics, and other table properties. The
+        only indication that it is not an HDFS table is the <codeph>s3a://</codeph> URL in the
+        <codeph>LOCATION</codeph> property. Many data files can reside in the S3 directory, and their combined
+        contents form the table data. Because the data in this example is uploaded after the table is created, a
+        <codeph>REFRESH</codeph> statement prompts Impala to update its cached information about the data files.
+      </p>
+
+<codeblock>[localhost:21000] > create table usa_cities_s3 like usa_cities location 's3a://impala-demo/usa_cities';
+[localhost:21000] > desc usa_cities_s3;
++-------+----------+---------+
+| name  | type     | comment |
++-------+----------+---------+
+| id    | smallint |         |
+| city  | string   |         |
+| state | string   |         |
++-------+----------+---------+
+
+-- Now from a web browser, upload the same data file(s) to S3 as in the HDFS table,
+-- under the relevant bucket and path. If you already have the data in S3, you would
+-- point the table LOCATION at an existing path.
+
+[localhost:21000] > refresh usa_cities_s3;
+[localhost:21000] > select count(*) from usa_cities_s3;
++----------+
+| count(*) |
++----------+
+| 289      |
++----------+
+[localhost:21000] > select distinct state from sample_data_s3 limit 5;
++----------------------+
+| state                |
++----------------------+
+| Louisiana            |
+| Minnesota            |
+| Georgia              |
+| Alaska               |
+| Ohio                 |
++----------------------+
+[localhost:21000] > desc formatted usa_cities_s3;
++------------------------------+------------------------------+---------+
+| name                         | type                         | comment |
++------------------------------+------------------------------+---------+
+| # col_name                   | data_type                    | comment |
+|                              | NULL                         | NULL    |
+| id                           | smallint                     | NULL    |
+| city                         | string                       | NULL    |
+| state                        | string                       | NULL    |
+|                              | NULL                         | NULL    |
+| # Detailed Table Information | NULL                         | NULL    |
+| Database:                    | s3_testing                   | NULL    |
+| Owner:                       | jrussell                     | NULL    |
+| CreateTime:                  | Mon Mar 16 11:36:25 PDT 2015 | NULL    |
+| LastAccessTime:              | UNKNOWN                      | NULL    |
+| Protect Mode:                | None                         | NULL    |
+| Retention:                   | 0                            | NULL    |
+| Location:                    | s3a://impala-demo/usa_cities | NULL    |
+| Table Type:                  | MANAGED_TABLE                | NULL    |
+...
++------------------------------+------------------------------+---------+
+</codeblock>
+
+<!-- Cut out unnecessary output, makes the example too wide.
+| Table Parameters:            | NULL                                                           | NULL                 |
+|                              | COLUMN_STATS_ACCURATE                                          | false                |
+|                              | numFiles                                                       | 0                    |
+|                              | numRows                                                        | -1                   |
+|                              | rawDataSize                                                    | -1                   |
+|                              | totalSize                                                      | 0                    |
+|                              | transient_lastDdlTime                                          | 1426528176           |
+|                              | NULL                                                           | NULL                 |
+| # Storage Information        | NULL                                                           | NULL                 |
+| SerDe Library:               | org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe             | NULL                 |
+| InputFormat:                 | org.apache.hadoop.mapred.TextInputFormat                       | NULL                 |
+| OutputFormat:                | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat     | NULL                 |
+| Compressed:                  | No                                                             | NULL                 |
+| Num Buckets:                 | 0                                                              | NULL                 |
+| Bucket Columns:              | []                                                             | NULL                 |
+| Sort Columns:                | []                                                             | NULL                 |
+-->
+
+      <p>
+        In this case, we have already uploaded a Parquet file with a million rows of data to the
+        <codeph>sample_data</codeph> directory underneath the <codeph>impala-demo</codeph> bucket on S3. This
+        session creates a table with matching column settings pointing to the corresponding location in S3, then
+        queries the table. Because the data is already in place on S3 when the table is created, no
+        <codeph>REFRESH</codeph> statement is required.
+      </p>
+
+<codeblock>[localhost:21000] > create table sample_data_s3
+                  > (id int, id bigint, val int, zerofill string,
+                  > name string, assertion boolean, city string, state string)
+                  > stored as parquet location 's3a://impala-demo/sample_data';
+[localhost:21000] > select count(*) from sample_data_s3;;
++----------+
+| count(*) |
++----------+
+| 1000000  |
++----------+
+[localhost:21000] > select count(*) howmany, assertion from sample_data_s3 group by assertion;
++---------+-----------+
+| howmany | assertion |
++---------+-----------+
+| 667149  | true      |
+| 332851  | false     |
++---------+-----------+
+</codeblock>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="s3_queries">
+
+    <title>Running and Tuning Impala Queries for Data Stored on S3</title>
+
+    <conbody>
+
+      <p>
+        Once the appropriate <codeph>LOCATION</codeph> attributes are set up at the table or partition level, you
+        query data stored in S3 exactly the same as data stored on HDFS or in HBase:
+      </p>
+
+      <ul>
+        <li>
+          Queries against S3 data support all the same file formats as for HDFS data.
+        </li>
+
+        <li>
+          Tables can be unpartitioned or partitioned. For partitioned tables, either manually construct paths in S3
+          corresponding to the HDFS directories representing partition key values, or use <codeph>ALTER TABLE ...
+          ADD PARTITION</codeph> to set up the appropriate paths in S3.
+        </li>
+
+        <li>
+          HDFS and HBase tables can be joined to S3 tables, or S3 tables can be joined with each other.
+        </li>
+
+        <li>
+          Authorization using the Sentry framework to control access to databases, tables, or columns works the
+          same whether the data is in HDFS or in S3.
+        </li>
+
+        <li>
+          The <cmdname>catalogd</cmdname> daemon caches metadata for both HDFS and S3 tables. Use
+          <codeph>REFRESH</codeph> and <codeph>INVALIDATE METADATA</codeph> for S3 tables in the same situations
+          where you would issue those statements for HDFS tables.
+        </li>
+
+        <li>
+          Queries against S3 tables are subject to the same kinds of admission control and resource management as
+          HDFS tables.
+        </li>
+
+        <li>
+          Metadata about S3 tables is stored in the same metastore database as for HDFS tables.
+        </li>
+
+        <li>
+          You can set up views referring to S3 tables, the same as for HDFS tables.
+        </li>
+
+        <li>
+          The <codeph>COMPUTE STATS</codeph>, <codeph>SHOW TABLE STATS</codeph>, and <codeph>SHOW COLUMN
+          STATS</codeph> statements work for S3 tables also.
+        </li>
+      </ul>
+
+    </conbody>
+
+    <concept id="s3_performance">
+
+      <title>Understanding and Tuning Impala Query Performance for S3 Data</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Performance"/>
+    </metadata>
+  </prolog>
+
+      <conbody>
+
+        <p>
+          Although Impala queries for data stored in S3 might be less performant than queries against the
+          equivalent data stored in HDFS, you can still do some tuning. Here are techniques you can use to
+          interpret explain plans and profiles for queries against S3 data, and tips to achieve the best
+          performance possible for such queries.
+        </p>
+
+        <p>
+          All else being equal, performance is expected to be lower for queries running against data on S3 rather
+          than HDFS. The actual mechanics of the <codeph>SELECT</codeph> statement are somewhat different when the
+          data is in S3. Although the work is still distributed across the datanodes of the cluster, Impala might
+          parallelize the work for a distributed query differently for data on HDFS and S3. S3 does not have the
+          same block notion as HDFS, so Impala uses heuristics to determine how to split up large S3 files for
+          processing in parallel. Because all hosts can access any S3 data file with equal efficiency, the
+          distribution of work might be different than for HDFS data, where the data blocks are physically read
+          using short-circuit local reads by hosts that contain the appropriate block replicas. Although the I/O to
+          read the S3 data might be spread evenly across the hosts of the cluster, the fact that all data is
+          initially retrieved across the network means that the overall query performance is likely to be lower for
+          S3 data than for HDFS data.
+        </p>
+
+        <p conref="../shared/impala_common.xml#common/s3_block_splitting"/>
+
+        <p conref="../shared/impala_common.xml#common/s3_dml_performance"/>
+
+        <p>
+          When optimizing aspects of for complex queries such as the join order, Impala treats tables on HDFS and
+          S3 the same way. Therefore, follow all the same tuning recommendations for S3 tables as for HDFS ones,
+          such as using the <codeph>COMPUTE STATS</codeph> statement to help Impala construct accurate estimates of
+          row counts and cardinality. See <xref href="impala_performance.xml#performance"/> for details.
+        </p>
+
+        <p>
+          In query profile reports, the numbers for <codeph>BytesReadLocal</codeph>,
+          <codeph>BytesReadShortCircuit</codeph>, <codeph>BytesReadDataNodeCached</codeph>, and
+          <codeph>BytesReadRemoteUnexpected</codeph> are blank because those metrics come from HDFS.
+          If you do see any indications that a query against an S3 table performed <q>remote read</q>
+          operations, do not be alarmed. That is expected because, by definition, all the I/O for S3 tables involves
+          remote reads.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+  </concept>
+
+  <concept id="s3_restrictions">
+
+    <title>Restrictions on Impala Support for S3</title>
+
+    <conbody>
+
+      <p>
+        Impala requires that the default filesystem for the cluster be HDFS. You cannot use S3 as the only
+        filesystem in the cluster.
+      </p>
+
+      <p rev="2.6.0 CDH-39913 IMPALA-1878">
+        Prior to <keyword keyref="impala26_full"/> Impala could not perform DML operations (<codeph>INSERT</codeph>,
+        <codeph>LOAD DATA</codeph>, or <codeph>CREATE TABLE AS SELECT</codeph>) where the destination is a table
+        or partition located on an S3 filesystem. This restriction is lifted in <keyword keyref="impala26_full"/> and higher.
+      </p>
+
+      <p>
+        Impala does not support the old <codeph>s3://</codeph> block-based and <codeph>s3n://</codeph> filesystem
+        schemes, only <codeph>s3a://</codeph>.
+      </p>
+
+      <p>
+        Although S3 is often used to store JSON-formatted data, the current Impala support for S3 does not include
+        directly querying JSON data. For Impala queries, use data files in one of the file formats listed in
+        <xref href="impala_file_formats.xml#file_formats"/>. If you have data in JSON format, you can prepare a
+        flattened version of that data for querying by Impala as part of your ETL cycle.
+      </p>
+
+      <p>
+        You cannot use the <codeph>ALTER TABLE ... SET CACHED</codeph> statement for tables or partitions that are
+        located in S3.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="s3_best_practices" rev="2.6.0 CDH-33310 CDH-39913 IMPALA-1878">
+    <title>Best Practices for Using Impala with S3</title>
+    <prolog>
+      <metadata>
+        <data name="Category" value="Guidelines"/>
+        <data name="Category" value="Best Practices"/>
+      </metadata>
+    </prolog>
+    <conbody>
+      <p>
+        The following guidelines represent best practices derived from testing and field experience with Impala on S3:
+      </p>
+      <ul>
+        <li>
+          <p>
+            Any reference to an S3 location must be fully qualified. (This rule applies when
+            S3 is not designated as the default filesystem.)
+          </p>
+        </li>
+        <li>
+          <p>
+            Set the safety valve <codeph>fs.s3a.connection.maximum</codeph> to 1500 for <cmdname>impalad</cmdname>.
+          </p>
+        </li>
+        <li>
+          <p>
+            Set safety valve <codeph>fs.s3a.block.size</codeph> to 134217728
+            (128 MB in bytes) if most Parquet files queried by Impala were written by Hive
+            or ParquetMR jobs. Set the block size to 268435456 (256 MB in bytes) if most Parquet
+            files queried by Impala were written by Impala.
+          </p>
+        </li>
+        <li>
+          <p>
+            <codeph>DROP TABLE .. PURGE</codeph> is much faster than the default <codeph>DROP TABLE</codeph>.
+            The same applies to <codeph>ALTER TABLE ... DROP PARTITION PURGE</codeph>
+            versus the default <codeph>DROP PARTITION</codeph> operation.
+            However, due to the eventually consistent nature of S3, the files for that
+            table or partition could remain for some unbounded time when using <codeph>PURGE</codeph>.
+            The default <codeph>DROP TABLE/PARTITION</codeph> is slow because Impala copies the files to the HDFS trash folder,
+            and Impala waits until all the data is moved. <codeph>DROP TABLE/PARTITION .. PURGE</codeph> is a
+            fast delete operation, and the Impala statement finishes quickly even though the change might not
+            have propagated fully throughout S3.
+          </p>
+        </li>
+        <li>
+          <p>
+            <codeph>INSERT</codeph> statements are faster than <codeph>INSERT OVERWRITE</codeph> for S3.
+            The query option <codeph>S3_SKIP_INSERT_STAGING</codeph>, which is set to <codeph>true</codeph> by default,
+            skips the staging step for regular <codeph>INSERT</codeph> (but not <codeph>INSERT OVERWRITE</codeph>).
+            This makes the operation much faster, but consistency is not guaranteed: if a node fails during execution, the
+            table could end up with inconsistent data. Set this option to <codeph>false</codeph> if stronger
+            consistency is required, however this setting will make the <codeph>INSERT</codeph> operations slower.
+          </p>
+        </li>
+        <li>
+          <p>
+            Too many files in a table can make metadata loading and updating slow on S3.
+            If too many requests are made to S3, S3 has a back-off mechanism and
+            responds slower than usual. You might have many small files because of:
+          </p>
+          <ul>
+            <li>
+              <p>
+                Too many partitions due to over-granular partitioning. Prefer partitions with
+                many megabytes of data, so that even a query against a single partition can
+                be parallelized effectively.
+              </p>
+            </li>
+            <li>
+              <p>
+                Many small <codeph>INSERT</codeph> queries. Prefer bulk
+                <codeph>INSERT</codeph>s so that more data is written to fewer
+                files.
+              </p>
+            </li>
+          </ul>
+        </li>
+      </ul>
+
+    </conbody>
+  </concept>
+
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_s3_skip_insert_staging.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_s3_skip_insert_staging.xml b/docs/topics/impala_s3_skip_insert_staging.xml
new file mode 100644
index 0000000..f9035a2
--- /dev/null
+++ b/docs/topics/impala_s3_skip_insert_staging.xml
@@ -0,0 +1,77 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="s3_skip_insert_staging" rev="2.6.0 IMPALA-3452 CDH-39913">
+
+  <title>S3_SKIP_INSERT_STAGING Query Option (<keyword keyref="impala26"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>S3_SKIP_INSERT_STAGING</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Amazon"/>
+      <data name="Category" value="S3"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.6.0 IMPALA-3452 CDH-39913">
+      <indexterm audience="Cloudera">IMPALA_S3_SKIP_INSERT_STAGING query option</indexterm>
+    </p>
+
+    <p>
+      Speeds up <codeph>INSERT</codeph> operations on tables or partitions residing on the
+      Amazon S3 filesystem. The tradeoff is the possibility of inconsistent data left behind
+      if an error occurs partway through the operation.
+    </p>
+
+    <p>
+      By default, Impala write operations to S3 tables and partitions involve a two-stage process.
+      Impala writes intermediate files to S3, then (because S3 does not provide a <q>rename</q>
+      operation) those intermediate files are copied to their final location, making the process
+      more expensive as on a filesystem that supports renaming or moving files.
+      This query option makes Impala skip the intermediate files, and instead write the
+      new data directly to the final destination.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <note type="important">
+      <p>
+        If a host that is participating in the <codeph>INSERT</codeph> operation fails partway through
+        the query, you might be left with a table or partition that contains some but not all of the
+        expected data files. Therefore, this option is most appropriate for a development or test
+        environment where you have the ability to reconstruct the table if a problem during
+        <codeph>INSERT</codeph> leaves the data in an inconsistent state.
+      </p>
+    </note>
+
+    <p>
+      The timing of file deletion during an <codeph>INSERT OVERWRITE</codeph> operation
+      makes it impractical to write new files to S3 and delete the old files in a single operation.
+      Therefore, this query option only affects regular <codeph>INSERT</codeph> statements that add
+      to the existing data in a table, not <codeph>INSERT OVERWRITE</codeph> statements.
+      Use <codeph>TRUNCATE TABLE</codeph> if you need to remove all contents from an S3 table
+      before performing a fast <codeph>INSERT</codeph> with this option enabled.
+    </p>
+
+    <p>
+      Performance improvements with this option enabled can be substantial. The speed increase
+      might be more noticeable for non-partitioned tables than for partitioned tables.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_boolean"/>
+    <p conref="../shared/impala_common.xml#common/default_true_1"/>
+
+    <p conref="../shared/impala_common.xml#common/added_in_260"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_s3.xml#s3"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_scalability.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_scalability.xml b/docs/topics/impala_scalability.xml
new file mode 100644
index 0000000..ea0d0a4
--- /dev/null
+++ b/docs/topics/impala_scalability.xml
@@ -0,0 +1,833 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="scalability">
+
+  <title>Scalability Considerations for Impala</title>
+  <titlealts audience="PDF"><navtitle>Scalability Considerations</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Planning"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Memory"/>
+      <data name="Category" value="Scalability"/>
+      <!-- Using domain knowledge about Impala, sizing, etc. to decide what to mark as 'Proof of Concept'. -->
+      <data name="Category" value="Proof of Concept"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      This section explains how the size of your cluster and the volume of data influences SQL performance and
+      schema design for Impala tables. Typically, adding more cluster capacity reduces problems due to memory
+      limits or disk throughput. On the other hand, larger clusters are more likely to have other kinds of
+      scalability issues, such as a single slow node that causes performance problems for queries.
+    </p>
+
+    <p outputclass="toc inpage"/>
+
+    <p conref="../shared/impala_common.xml#common/cookbook_blurb"/>
+
+  </conbody>
+
+  <concept audience="Cloudera" id="scalability_memory">
+
+    <title>Overview and Guidelines for Impala Memory Usage</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Memory"/>
+      <data name="Category" value="Concepts"/>
+      <data name="Category" value="Best Practices"/>
+      <data name="Category" value="Guidelines"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+<!--
+Outline adapted from Alan Choi's "best practices" and/or "performance cookbook" papers.
+-->
+
+<codeblock>Memory Usage \u2013 the Basics
+*  Memory is used by:
+*  Hash join \u2013 RHS tables after decompression, filtering and projection
+*  Group by \u2013 proportional to the #groups
+*  Parquet writer buffer \u2013 1GB per partition
+*  IO buffer (shared across queries)
+*  Metadata cache (no more than 1GB typically)
+*  Memory held and reused by later query
+*  Impala releases memory from time to time starting in 1.4.
+
+Memory Usage \u2013 Estimating Memory Usage
+*  Use Explain Plan
+* Requires statistics! Mem estimate without stats is meaningless.
+* Reports per-host memory requirement for this cluster size.
+*  Re-run if you\u2019ve re-sized the cluster!
+[image of explain plan]
+
+Memory Usage \u2013 Estimating Memory Usage
+*  EXPLAIN\u2019s memory estimate issues
+*  Can be way off \u2013 much higher or much lower.
+*  group by\u2019s estimate can be particularly off \u2013 when there\u2019s a large number of group by columns.
+*  Mem estimate = NDV of group by column 1 * NDV of group by column 2 * ... NDV of group by column n
+*  Ignore EXPLAIN\u2019s estimate if it\u2019s too high! \u2022  Do your own estimate for group by
+*  GROUP BY mem usage = (total number of groups * size of each row) + (total number of groups * size of each row) / num node
+
+Memory Usage \u2013 Finding Actual Memory Usage
+*  Search for \u201cPer Node Peak Memory Usage\u201d in the profile.
+This is accurate. Use it for production capacity planning.
+
+Memory Usage \u2013 Actual Memory Usage
+*  For complex queries, how do I know which part of my query is using too much memory?
+*  Use the ExecSummary from the query profile!
+- But is that "Peak Mem" number aggregate or per-node?
+[image of executive summary]
+
+Memory Usage \u2013 Hitting Mem-limit
+*  Top causes (in order) of hitting mem-limit even when running a single query:
+1. Lack of statistics
+2. Lots of joins within a single query
+3. Big-table joining big-table
+4. Gigantic group by
+
+Memory Usage \u2013 Hitting Mem-limit
+Lack of stats
+*  Wrong join order, wrong join strategy, wrong insert strategy
+*  Explain Plan tells you that!
+[image of explain plan]
+*  Fix: Compute Stats table
+
+Memory Usage \u2013 Hitting Mem-limit
+Lots of joins within a single query
+* select...from fact, dim1, dim2,dim3,...dimN where ...
+* Each dim tbl can fit in memory, but not all of them together
+* As of Impala 1.4, Impala might choose the wrong plan \u2013 BROADCAST
+FIX 1: use shuffle hint
+select ... from fact join [shuffle] dim1 on ... join dim2 [shuffle] ...
+FIX 2: pre-join the dim tables (if possible)
+- How about an example to illustrate that technique?
+* few join=&gt;better perf!
+
+Memory Usage: Hitting Mem-limit
+Big-table joining big-table
+*  Big-table (after decompression, filtering, and projection) is a table that is bigger than total cluster memory size.
+*  Impala 2.0 will do this (via disk-based join). Consider using Hive for now.
+*  (Advanced) For a simple query, you can try this advanced workaround \u2013 per-partition join
+*  Requires the partition key be part of the join key
+select ... from BigTbl_A a join BigTbl_B b where a.part_key = b.part_key and a.part_key in (1,2,3)
+   union all
+select ... from BigTbl_A a join BigTbl_B b where a.part_key = b.part_key and a.part_key in (4,5,6)
+
+Memory Usage: Hitting Mem-limit
+Gigantic group by
+* The total number of distinct groups is huge, such as group by userid.
+* Impala 2.0 will do this (via disk-based agg). Consider using Hive for now.
+- Is this one of the cases where people were unhappy we recommended Hive?
+* (Advanced) For a simple query, you can try this advanced workaround \u2013 per-partition agg
+*  Requires the partition key be part of the group by
+select part_key, col1, col2, ...agg(..) from tbl where
+       part_key in (1,2,3)
+       Union all
+       Select part_key, col1, col2, ...agg(..) from tbl where
+       part_key in (4,5,6)
+- But where's the GROUP BY in the preceding query? Need a real example.
+
+Memory Usage: Additional Notes
+*  Use explain plan for estimate; use profile for accurate measure
+*  Data skew can use uneven memory usage
+*  Review previous common issues on out-of-memory
+*  Note: Even with disk-based joins, you'll want to review these steps to speed up queries and use memory more efficiently
+</codeblock>
+    </conbody>
+  </concept>
+
+  <concept id="scalability_catalog">
+
+    <title>Impact of Many Tables or Partitions on Impala Catalog Performance and Memory Usage</title>
+
+    <conbody>
+
+      <p audience="Cloudera">
+        Details to fill in in future: Impact of <q>load catalog in background</q> option.
+        Changing timeouts. Related Cloudera Manager settings.
+      </p>
+
+      <p>
+        Because Hadoop I/O is optimized for reading and writing large files, Impala is optimized for tables
+        containing relatively few, large data files. Schemas containing thousands of tables, or tables containing
+        thousands of partitions, can encounter performance issues during startup or during DDL operations such as
+        <codeph>ALTER TABLE</codeph> statements.
+      </p>
+
+      <note type="important" rev="TSB-168">
+      <p>
+        Because of a change in the default heap size for the <cmdname>catalogd</cmdname> daemon in
+        <keyword keyref="impala25_full"/> and higher, the following procedure to increase the <cmdname>catalogd</cmdname>
+        memory limit might be required following an upgrade to <keyword keyref="impala25_full"/> even if not
+        needed previously.
+      </p>
+      </note>
+
+      <p conref="../shared/impala_common.xml#common/increase_catalogd_heap_size"/>
+
+    </conbody>
+  </concept>
+
+  <concept rev="2.1.0" id="statestore_scalability">
+
+    <title>Scalability Considerations for the Impala Statestore</title>
+
+    <conbody>
+
+      <p>
+        Before CDH 5.3, the statestore sent only one kind of message to its subscribers. This message contained all
+        updates for any topics that a subscriber had subscribed to. It also served to let subscribers know that the
+        statestore had not failed, and conversely the statestore used the success of sending a heartbeat to a
+        subscriber to decide whether or not the subscriber had failed.
+      </p>
+
+      <p>
+        Combining topic updates and failure detection in a single message led to bottlenecks in clusters with large
+        numbers of tables, partitions, and HDFS data blocks. When the statestore was overloaded with metadata
+        updates to transmit, heartbeat messages were sent less frequently, sometimes causing subscribers to time
+        out their connection with the statestore. Increasing the subscriber timeout and decreasing the frequency of
+        statestore heartbeats worked around the problem, but reduced responsiveness when the statestore failed or
+        restarted.
+      </p>
+
+      <p>
+        As of CDH 5.3, the statestore now sends topic updates and heartbeats in separate messages. This allows the
+        statestore to send and receive a steady stream of lightweight heartbeats, and removes the requirement to
+        send topic updates according to a fixed schedule, reducing statestore network overhead.
+      </p>
+
+      <p>
+        The statestore now has the following relevant configuration flags for the <cmdname>statestored</cmdname>
+        daemon:
+      </p>
+
+      <dl>
+        <dlentry id="statestore_num_update_threads">
+
+          <dt>
+            <codeph>-statestore_num_update_threads</codeph>
+          </dt>
+
+          <dd>
+            The number of threads inside the statestore dedicated to sending topic updates. You should not
+            typically need to change this value.
+            <p>
+              <b>Default:</b> 10
+            </p>
+          </dd>
+
+        </dlentry>
+
+        <dlentry id="statestore_update_frequency_ms">
+
+          <dt>
+            <codeph>-statestore_update_frequency_ms</codeph>
+          </dt>
+
+          <dd>
+            The frequency, in milliseconds, with which the statestore tries to send topic updates to each
+            subscriber. This is a best-effort value; if the statestore is unable to meet this frequency, it sends
+            topic updates as fast as it can. You should not typically need to change this value.
+            <p>
+              <b>Default:</b> 2000
+            </p>
+          </dd>
+
+        </dlentry>
+
+        <dlentry id="statestore_num_heartbeat_threads">
+
+          <dt>
+            <codeph>-statestore_num_heartbeat_threads</codeph>
+          </dt>
+
+          <dd>
+            The number of threads inside the statestore dedicated to sending heartbeats. You should not typically
+            need to change this value.
+            <p>
+              <b>Default:</b> 10
+            </p>
+          </dd>
+
+        </dlentry>
+
+        <dlentry id="statestore_heartbeat_frequency_ms">
+
+          <dt>
+            <codeph>-statestore_heartbeat_frequency_ms</codeph>
+          </dt>
+
+          <dd>
+            The frequency, in milliseconds, with which the statestore tries to send heartbeats to each subscriber.
+            This value should be good for large catalogs and clusters up to approximately 150 nodes. Beyond that,
+            you might need to increase this value to make the interval longer between heartbeat messages.
+            <p>
+              <b>Default:</b> 1000 (one heartbeat message every second)
+            </p>
+          </dd>
+
+        </dlentry>
+      </dl>
+
+      <p>
+        As of CDH 5.3, not all of these flags are present in the Cloudera Manager user interface. Some must be set
+        using the <uicontrol>Advanced Configuration Snippet</uicontrol> fields for the statestore component.
+      </p>
+
+      <p>
+        If it takes a very long time for a cluster to start up, and <cmdname>impala-shell</cmdname> consistently
+        displays <codeph>This Impala daemon is not ready to accept user requests</codeph>, the statestore might be
+        taking too long to send the entire catalog topic to the cluster. In this case, consider adding
+        <codeph>--load_catalog_in_background=false</codeph> to your catalog service configuration. This setting
+        stops the statestore from loading the entire catalog into memory at cluster startup. Instead, metadata for
+        each table is loaded when the table is accessed for the first time.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept audience="Cloudera" id="scalability_cluster_size">
+
+    <title>Scalability Considerations for Impala Cluster Size and Topology</title>
+
+    <conbody>
+
+      <p>
+      </p>
+    </conbody>
+  </concept>
+
+  <concept audience="Cloudera" id="concurrent_connections">
+
+    <title>Scaling the Number of Concurrent Connections</title>
+
+    <conbody>
+
+      <p></p>
+    </conbody>
+  </concept>
+
+  <concept rev="2.0.0" id="spill_to_disk">
+
+    <title>SQL Operations that Spill to Disk</title>
+
+    <conbody>
+
+      <p>
+        Certain memory-intensive operations write temporary data to disk (known as <term>spilling</term> to disk)
+        when Impala is close to exceeding its memory limit on a particular host.
+      </p>
+
+      <p>
+        The result is a query that completes successfully, rather than failing with an out-of-memory error. The
+        tradeoff is decreased performance due to the extra disk I/O to write the temporary data and read it back
+        in. The slowdown could be potentially be significant. Thus, while this feature improves reliability,
+        you should optimize your queries, system parameters, and hardware configuration to make this spilling a rare occurrence.
+      </p>
+
+      <p>
+        <b>What kinds of queries might spill to disk:</b>
+      </p>
+
+      <p>
+        Several SQL clauses and constructs require memory allocations that could activat the spilling mechanism:
+      </p>
+      <ul>
+        <li>
+          <p>
+            when a query uses a <codeph>GROUP BY</codeph> clause for columns
+            with millions or billions of distinct values, Impala keeps a
+            similar number of temporary results in memory, to accumulate the
+            aggregate results for each value in the group.
+          </p>
+        </li>
+        <li>
+          <p>
+            When large tables are joined together, Impala keeps the values of
+            the join columns from one table in memory, to compare them to
+            incoming values from the other table.
+          </p>
+        </li>
+        <li>
+          <p>
+            When a large result set is sorted by the <codeph>ORDER BY</codeph>
+            clause, each node sorts its portion of the result set in memory.
+          </p>
+        </li>
+        <li>
+          <p>
+            The <codeph>DISTINCT</codeph> and <codeph>UNION</codeph> operators
+            build in-memory data structures to represent all values found so
+            far, to eliminate duplicates as the query progresses.
+          </p>
+        </li>
+        <!-- JIRA still in open state as of 5.8 / 2.6, commenting out.
+        <li>
+          <p rev="IMPALA-3471">
+            In <keyword keyref="impala26_full"/> and higher, <term>top-N</term> queries (those with
+            <codeph>ORDER BY</codeph> and <codeph>LIMIT</codeph> clauses) can also spill.
+            Impala allocates enough memory to hold as many rows as specified by the <codeph>LIMIT</codeph>
+            clause, plus enough memory to hold as many rows as specified by any <codeph>OFFSET</codeph> clause.
+          </p>
+        </li>
+        -->
+      </ul>
+
+      <p conref="../shared/impala_common.xml#common/spill_to_disk_vs_dynamic_partition_pruning"/>
+
+      <p>
+        <b>How Impala handles scratch disk space for spilling:</b>
+      </p>
+
+      <p rev="obwl" conref="../shared/impala_common.xml#common/order_by_scratch_dir"/>
+
+      <p>
+        <b>Memory usage for SQL operators:</b>
+      </p>
+
+      <p>
+        The infrastructure of the spilling feature affects the way the affected SQL operators, such as
+        <codeph>GROUP BY</codeph>, <codeph>DISTINCT</codeph>, and joins, use memory.
+        On each host that participates in the query, each such operator in a query accumulates memory
+        while building the data structure to process the aggregation or join operation. The amount
+        of memory used depends on the portion of the data being handled by that host, and thus might
+        be different from one host to another. When the amount of memory being used for the operator
+        on a particular host reaches a threshold amount, Impala reserves an additional memory buffer
+        to use as a work area in case that operator causes the query to exceed the memory limit for
+        that host. After allocating the memory buffer, the memory used by that operator remains
+        essentially stable or grows only slowly, until the point where the memory limit is reached
+        and the query begins writing temporary data to disk.
+      </p>
+
+      <p rev="2.2.0">
+        Prior to Impala 2.2 (CDH 5.4), the extra memory buffer for an operator that might spill to disk
+        was allocated when the data structure used by the applicable SQL operator reaches 16 MB in size,
+        and the memory buffer itself was 512 MB. In Impala 2.2, these values are halved: the threshold value
+        is 8 MB and the memory buffer is 256 MB. <ph rev="2.3.0">In <keyword keyref="impala23_full"/> and higher, the memory for the buffer
+        is allocated in pieces, only as needed, to avoid sudden large jumps in memory usage.</ph> A query that uses
+        multiple such operators might allocate multiple such memory buffers, as the size of the data structure
+        for each operator crosses the threshold on a particular host.
+      </p>
+
+      <p>
+        Therefore, a query that processes a relatively small amount of data on each host would likely
+        never reach the threshold for any operator, and would never allocate any extra memory buffers. A query
+        that did process millions of groups, distinct values, join keys, and so on might cross the threshold,
+        causing its memory requirement to rise suddenly and then flatten out. The larger the cluster, less data is processed
+        on any particular host, thus reducing the chance of requiring the extra memory allocation.
+      </p>
+
+      <p>
+        <b>Added in:</b> This feature was added to the <codeph>ORDER BY</codeph> clause in Impala 1.4 for CDH 4,
+        and in CDH 5.1. This feature was extended to cover join queries, aggregation functions, and analytic
+        functions in Impala 2.0 for CDH 4, and in CDH 5.2. The size of the memory work area required by
+        each operator that spills was reduced from 512 megabytes to 256 megabytes in Impala 2.2 (CDH 5.4).
+      </p>
+
+      <p>
+        <b>Avoiding queries that spill to disk:</b>
+      </p>
+
+      <p>
+        Because the extra I/O can impose significant performance overhead on these types of queries, try to avoid
+        this situation by using the following steps:
+      </p>
+
+      <ol>
+        <li>
+          Detect how often queries spill to disk, and how much temporary data is written. Refer to the following
+          sources:
+          <ul>
+            <li>
+              The output of the <codeph>PROFILE</codeph> command in the <cmdname>impala-shell</cmdname>
+              interpreter. This data shows the memory usage for each host and in total across the cluster. The
+              <codeph>BlockMgr.BytesWritten</codeph> counter reports how much data was written to disk during the
+              query.
+            </li>
+
+            <li>
+              The <uicontrol>Impala Queries</uicontrol> dialog in Cloudera Manager. You can see the peak memory
+              usage for a query, combined across all nodes in the cluster.
+            </li>
+
+            <li>
+              The <uicontrol>Queries</uicontrol> tab in the Impala debug web user interface. Select the query to
+              examine and click the corresponding <uicontrol>Profile</uicontrol> link. This data breaks down the
+              memory usage for a single host within the cluster, the host whose web interface you are connected to.
+            </li>
+          </ul>
+        </li>
+
+        <li>
+          Use one or more techniques to reduce the possibility of the queries spilling to disk:
+          <ul>
+            <li>
+              Increase the Impala memory limit if practical, for example, if you can increase the available memory
+              by more than the amount of temporary data written to disk on a particular node. Remember that in
+              Impala 2.0 and later, you can issue <codeph>SET MEM_LIMIT</codeph> as a SQL statement, which lets you
+              fine-tune the memory usage for queries from JDBC and ODBC applications.
+            </li>
+
+            <li>
+              Increase the number of nodes in the cluster, to increase the aggregate memory available to Impala and
+              reduce the amount of memory required on each node.
+            </li>
+
+            <li>
+              Increase the overall memory capacity of each DataNode at the hardware level.
+            </li>
+
+            <li>
+              On a cluster with resources shared between Impala and other Hadoop components, use resource
+              management features to allocate more memory for Impala. See
+              <xref href="impala_resource_management.xml#resource_management"/> for details.
+            </li>
+
+            <li>
+              If the memory pressure is due to running many concurrent queries rather than a few memory-intensive
+              ones, consider using the Impala admission control feature to lower the limit on the number of
+              concurrent queries. By spacing out the most resource-intensive queries, you can avoid spikes in
+              memory usage and improve overall response times. See
+              <xref href="impala_admission.xml#admission_control"/> for details.
+            </li>
+
+            <li>
+              Tune the queries with the highest memory requirements, using one or more of the following techniques:
+              <ul>
+                <li>
+                  Run the <codeph>COMPUTE STATS</codeph> statement for all tables involved in large-scale joins and
+                  aggregation queries.
+                </li>
+
+                <li>
+                  Minimize your use of <codeph>STRING</codeph> columns in join columns. Prefer numeric values
+                  instead.
+                </li>
+
+                <li>
+                  Examine the <codeph>EXPLAIN</codeph> plan to understand the execution strategy being used for the
+                  most resource-intensive queries. See <xref href="impala_explain_plan.xml#perf_explain"/> for
+                  details.
+                </li>
+
+                <li>
+                  If Impala still chooses a suboptimal execution strategy even with statistics available, or if it
+                  is impractical to keep the statistics up to date for huge or rapidly changing tables, add hints
+                  to the most resource-intensive queries to select the right execution strategy. See
+                  <xref href="impala_hints.xml#hints"/> for details.
+                </li>
+              </ul>
+            </li>
+
+            <li>
+              If your queries experience substantial performance overhead due to spilling, enable the
+              <codeph>DISABLE_UNSAFE_SPILLS</codeph> query option. This option prevents queries whose memory usage
+              is likely to be exorbitant from spilling to disk. See
+              <xref href="impala_disable_unsafe_spills.xml#disable_unsafe_spills"/> for details. As you tune
+              problematic queries using the preceding steps, fewer and fewer will be cancelled by this option
+              setting.
+            </li>
+          </ul>
+        </li>
+      </ol>
+
+      <p>
+        <b>Testing performance implications of spilling to disk:</b>
+      </p>
+
+      <p>
+        To artificially provoke spilling, to test this feature and understand the performance implications, use a
+        test environment with a memory limit of at least 2 GB. Issue the <codeph>SET</codeph> command with no
+        arguments to check the current setting for the <codeph>MEM_LIMIT</codeph> query option. Set the query
+        option <codeph>DISABLE_UNSAFE_SPILLS=true</codeph>. This option limits the spill-to-disk feature to prevent
+        runaway disk usage from queries that are known in advance to be suboptimal. Within
+        <cmdname>impala-shell</cmdname>, run a query that you expect to be memory-intensive, based on the criteria
+        explained earlier. A self-join of a large table is a good candidate:
+      </p>
+
+<codeblock>select count(*) from big_table a join big_table b using (column_with_many_values);
+</codeblock>
+
+      <p>
+        Issue the <codeph>PROFILE</codeph> command to get a detailed breakdown of the memory usage on each node
+        during the query. The crucial part of the profile output concerning memory is the <codeph>BlockMgr</codeph>
+        portion. For example, this profile shows that the query did not quite exceed the memory limit.
+      </p>
+
+<codeblock>BlockMgr:
+   - BlockWritesIssued: 1
+   - BlockWritesOutstanding: 0
+   - BlocksCreated: 24
+   - BlocksRecycled: 1
+   - BufferedPins: 0
+   - MaxBlockSize: 8.00 MB (8388608)
+   <b>- MemoryLimit: 200.00 MB (209715200)</b>
+   <b>- PeakMemoryUsage: 192.22 MB (201555968)</b>
+   - TotalBufferWaitTime: 0ns
+   - TotalEncryptionTime: 0ns
+   - TotalIntegrityCheckTime: 0ns
+   - TotalReadBlockTime: 0ns
+</codeblock>
+
+      <p>
+        In this case, because the memory limit was already below any recommended value, I increased the volume of
+        data for the query rather than reducing the memory limit any further.
+      </p>
+
+      <p>
+        Set the <codeph>MEM_LIMIT</codeph> query option to a value that is smaller than the peak memory usage
+        reported in the profile output. Do not specify a memory limit lower than about 300 MB, because with such a
+        low limit, queries could fail to start for other reasons. Now try the memory-intensive query again.
+      </p>
+
+      <p>
+        Check if the query fails with a message like the following:
+      </p>
+
+<codeblock>WARNINGS: Spilling has been disabled for plans that do not have stats and are not hinted
+to prevent potentially bad plans from using too many cluster resources. Compute stats on
+these tables, hint the plan or disable this behavior via query options to enable spilling.
+</codeblock>
+
+      <p>
+        If so, the query could have consumed substantial temporary disk space, slowing down so much that it would
+        not complete in any reasonable time. Rather than rely on the spill-to-disk feature in this case, issue the
+        <codeph>COMPUTE STATS</codeph> statement for the table or tables in your sample query. Then run the query
+        again, check the peak memory usage again in the <codeph>PROFILE</codeph> output, and adjust the memory
+        limit again if necessary to be lower than the peak memory usage.
+      </p>
+
+      <p>
+        At this point, you have a query that is memory-intensive, but Impala can optimize it efficiently so that
+        the memory usage is not exorbitant. You have set an artificial constraint through the
+        <codeph>MEM_LIMIT</codeph> option so that the query would normally fail with an out-of-memory error. But
+        the automatic spill-to-disk feature means that the query should actually succeed, at the expense of some
+        extra disk I/O to read and write temporary work data.
+      </p>
+
+      <p>
+        Try the query again, and confirm that it succeeds. Examine the <codeph>PROFILE</codeph> output again. This
+        time, look for lines of this form:
+      </p>
+
+<codeblock>- SpilledPartitions: <varname>N</varname>
+</codeblock>
+
+      <p>
+        If you see any such lines with <varname>N</varname> greater than 0, that indicates the query would have
+        failed in Impala releases prior to 2.0, but now it succeeded because of the spill-to-disk feature. Examine
+        the total time taken by the <codeph>AGGREGATION_NODE</codeph> or other query fragments containing non-zero
+        <codeph>SpilledPartitions</codeph> values. Compare the times to similar fragments that did not spill, for
+        example in the <codeph>PROFILE</codeph> output when the same query is run with a higher memory limit. This
+        gives you an idea of the performance penalty of the spill operation for a particular query with a
+        particular memory limit. If you make the memory limit just a little lower than the peak memory usage, the
+        query only needs to write a small amount of temporary data to disk. The lower you set the memory limit, the
+        more temporary data is written and the slower the query becomes.
+      </p>
+
+      <p>
+        Now repeat this procedure for actual queries used in your environment. Use the
+        <codeph>DISABLE_UNSAFE_SPILLS</codeph> setting to identify cases where queries used more memory than
+        necessary due to lack of statistics on the relevant tables and columns, and issue <codeph>COMPUTE
+        STATS</codeph> where necessary.
+      </p>
+
+      <p>
+        <b>When to use DISABLE_UNSAFE_SPILLS:</b>
+      </p>
+
+      <p>
+        You might wonder, why not leave <codeph>DISABLE_UNSAFE_SPILLS</codeph> turned on all the time. Whether and
+        how frequently to use this option depends on your system environment and workload.
+      </p>
+
+      <p>
+        <codeph>DISABLE_UNSAFE_SPILLS</codeph> is suitable for an environment with ad hoc queries whose performance
+        characteristics and memory usage are not known in advance. It prevents <q>worst-case scenario</q> queries
+        that use large amounts of memory unnecessarily. Thus, you might turn this option on within a session while
+        developing new SQL code, even though it is turned off for existing applications.
+      </p>
+
+      <p>
+        Organizations where table and column statistics are generally up-to-date might leave this option turned on
+        all the time, again to avoid worst-case scenarios for untested queries or if a problem in the ETL pipeline
+        results in a table with no statistics. Turning on <codeph>DISABLE_UNSAFE_SPILLS</codeph> lets you <q>fail
+        fast</q> in this case and immediately gather statistics or tune the problematic queries.
+      </p>
+
+      <p>
+        Some organizations might leave this option turned off. For example, you might have tables large enough that
+        the <codeph>COMPUTE STATS</codeph> takes substantial time to run, making it impractical to re-run after
+        loading new data. If you have examined the <codeph>EXPLAIN</codeph> plans of your queries and know that
+        they are operating efficiently, you might leave <codeph>DISABLE_UNSAFE_SPILLS</codeph> turned off. In that
+        case, you know that any queries that spill will not go overboard with their memory consumption.
+      </p>
+
+<!--
+      <p>
+        <b>Turning off the spill-to-disk feature: (<keyword keyref="impala24_full"/> and lower only)</b>
+      </p>
+
+      <p>
+        Prior to <keyword keyref="impala25_full"/> certain conditions...
+      </p>
+
+      <p>
+        You might turn off the spill-to-disk feature if you are in an environment with constraints on disk space,
+        or if you prefer for queries that exceed the memory capacity in your cluster to <q>fail fast</q> so that
+        you can tune and retry them.
+      </p>
+
+      <p>
+        To turn off this feature, set the following configuration options for each <cmdname>impalad</cmdname>
+        daemon, either through the <cmdname>impalad</cmdname> advanced configuration snippet in Cloudera Manager,
+        or during <cmdname>impalad</cmdname> startup on each DataNode on systems not managed by Cloudera Manager:
+      </p>
+
+<codeblock>&minus;&minus;enable_partitioned_aggregation=false
+&minus;&minus;enable_partitioned_hash_join=false
+</codeblock>
+-->
+
+    </conbody>
+  </concept>
+
+<concept id="complex_query">
+<title>Limits on Query Size and Complexity</title>
+<conbody>
+<p>
+There are hardcoded limits on the maximum size and complexity of queries.
+Currently, the maximum number of expressions in a query is 2000.
+You might exceed the limits with large or deeply nested queries
+produced by business intelligence tools or other query generators.
+</p>
+<p>
+If you have the ability to customize such queries or the query generation
+logic that produces them, replace sequences of repetitive expressions
+with single operators such as <codeph>IN</codeph> or <codeph>BETWEEN</codeph>
+that can represent multiple values or ranges.
+For example, instead of a large number of <codeph>OR</codeph> clauses:
+</p>
+<codeblock>WHERE val = 1 OR val = 2 OR val = 6 OR val = 100 ...
+</codeblock>
+<p>
+use a single <codeph>IN</codeph> clause:
+</p>
+<codeblock>WHERE val IN (1,2,6,100,...)</codeblock>
+</conbody>
+</concept>
+
+<concept id="scalability_io">
+<title>Scalability Considerations for Impala I/O</title>
+<conbody>
+<p>
+Impala parallelizes its I/O operations aggressively,
+therefore the more disks you can attach to each host, the better.
+Impala retrieves data from disk so quickly using
+bulk read operations on large blocks, that most queries
+are CPU-bound rather than I/O-bound.
+</p>
+<p>
+Because the kind of sequential scanning typically done by
+Impala queries does not benefit much from the random-access
+capabilities of SSDs, spinning disks typically provide
+the most cost-effective kind of storage for Impala data,
+with little or no performance penalty as compared to SSDs.
+</p>
+<p>
+Resource management features such as YARN, Llama, and admission control
+typically constrain the amount of memory, CPU, or overall number of
+queries in a high-concurrency environment.
+Currently, there is no throttling mechanism for Impala I/O.
+</p>
+</conbody>
+</concept>
+
+<concept id="big_tables">
+<title>Scalability Considerations for Table Layout</title>
+<conbody>
+<p>
+Due to the overhead of retrieving and updating table metadata
+in the metastore database, try to limit the number of columns
+in a table to a maximum of approximately 2000.
+Although Impala can handle wider tables than this, the metastore overhead
+can become significant, leading to query performance that is slower
+than expected based on the actual data volume.
+</p>
+<p>
+To minimize overhead related to the metastore database and Impala query planning,
+try to limit the number of partitions for any partitioned table to a few tens of thousands.
+</p>
+</conbody>
+</concept>
+
+<concept rev="CDH-38321" id="kerberos_overhead_cluster_size">
+<title>Kerberos-Related Network Overhead for Large Clusters</title>
+<conbody>
+<p>
+When Impala starts up, or after each <codeph>kinit</codeph> refresh, Impala sends a number of
+simultaneous requests to the KDC. For a cluster with 100 hosts, the KDC might be able to process
+all the requests within roughly 5 seconds. For a cluster with 1000 hosts, the time to process
+the requests would be roughly 500 seconds. Impala also makes a number of DNS requests at the same
+time as these Kerberos-related requests.
+</p>
+<p>
+While these authentication requests are being processed, any submitted Impala queries will fail.
+During this period, the KDC and DNS may be slow to respond to requests from components other than Impala,
+so other secure services might be affected temporarily.
+</p>
+<p>
+To reduce the frequency of the <codeph>kinit</codeph> renewal that initiates a new set of
+authentication requests, increase the <codeph>kerberos_reinit_interval</codeph> configuration setting
+for the <cmdname>impalad</cmdname> daemons. Currently, the default for a cluster not managed by
+Cloudera Manager is 60 minutes, while the default under Cloudera Manager is 10 minutes.
+Consider using a higher value such as 360 (6 hours).
+</p>
+</conbody>
+</concept>
+
+  <concept id="scalability_hotspots" rev="2.5.0 IMPALA-2696">
+    <title>Avoiding CPU Hotspots for HDFS Cached Data</title>
+    <conbody>
+      <p>
+        You can use the HDFS caching feature, described in <xref href="impala_perf_hdfs_caching.xml#hdfs_caching"/>,
+        with Impala to reduce I/O and memory-to-memory copying for frequently accessed tables or partitions.
+      </p>
+      <p>
+        In the early days of this feature, you might have found that enabling HDFS caching
+        resulted in little or no performance improvement, because it could result in
+        <q>hotspots</q>: instead of the I/O to read the table data being parallelized across
+        the cluster, the I/O was reduced but the CPU load to process the data blocks
+        might be concentrated on a single host.
+      </p>
+      <p>
+        To avoid hotspots, include the <codeph>WITH REPLICATION</codeph> clause with the
+        <codeph>CREATE TABLE</codeph> or <codeph>ALTER TABLE</codeph> statements for tables that use HDFS caching.
+        This clause allows more than one host to cache the relevant data blocks, so the CPU load
+        can be shared, reducing the load on any one host.
+        See <xref href="impala_create_table.xml#create_table"/> and <xref href="impala_alter_table.xml#alter_table"/>
+        for details.
+      </p>
+      <p>
+        Hotspots with high CPU load for HDFS cached data could still arise in some cases, due to
+        the way that Impala schedules the work of processing data blocks on different hosts.
+        In <keyword keyref="impala25_full"/> and higher, scheduling improvements mean that the work for
+        HDFS cached data is divided better among all the hosts that have cached replicas
+        for a particular data block. When more than one host has a cached replica for a data block,
+        Impala assigns the work of processing that block to whichever host has done the least work
+        (in terms of number of bytes read) for the current query. If hotspots persist even with this
+        load-based scheduling algorithm, you can enable the query option <codeph>SCHEDULE_RANDOM_REPLICA=TRUE</codeph>
+        to further distribute the CPU load. This setting causes Impala to randomly pick a host to process a cached
+        data block if the scheduling algorithm encounters a tie when deciding which host has done the
+        least work.
+      </p>
+    </conbody>
+  </concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_scan_node_codegen_threshold.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_scan_node_codegen_threshold.xml b/docs/topics/impala_scan_node_codegen_threshold.xml
new file mode 100644
index 0000000..c81bbc6
--- /dev/null
+++ b/docs/topics/impala_scan_node_codegen_threshold.xml
@@ -0,0 +1,75 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="scan_node_codegen_threshold" rev="2.5.0 IMPALA-1755">
+
+  <title>SCAN_NODE_CODEGEN_THRESHOLD Query Option (<keyword keyref="impala25"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle></navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.5.0 IMPALA-1755">
+      <indexterm audience="Cloudera">SCAN_NODE_CODEGEN_THRESHOLD query option</indexterm>
+      The <codeph>SCAN_NODE_CODEGEN_THRESHOLD</codeph> query option
+      adjusts the aggressiveness of the code generation optimization process
+      when performing I/O read operations. It can help to work around performance problems
+      for queries where the table is small and the <codeph>WHERE</codeph> clause is complicated.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_integer"/>
+
+    <p>
+      <b>Default:</b> 1800000 (1.8 million)
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/added_in_250"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      This query option is intended mainly for the case where a query with a very complicated
+      <codeph>WHERE</codeph> clause, such as an <codeph>IN</codeph> operator with thousands
+      of entries, is run against a small table, especially a small table using Parquet format.
+      The code generation phase can become the dominant factor in the query response time,
+      making the query take several seconds even though there is relatively little work to do.
+      In this case, increase the value of this option to a much larger amount, anything up to
+      the maximum for a 32-bit integer.
+    </p>
+
+    <p>
+      Because this option only affects the code generation phase for the portion of the
+      query that performs I/O (the <term>scan nodes</term> within the query plan), it
+      lets you continue to keep code generation enabled for other queries, and other parts
+      of the same query, that can benefit from it. In contrast, the
+      <codeph>IMPALA_DISABLE_CODEGEN</codeph> query option turns off code generation entirely.
+    </p>
+
+    <p>
+      Because of the way the work for queries is divided internally, this option might not
+      affect code generation for all kinds of queries. If a plan fragment contains a scan
+      node and some other kind of plan node, code generation still occurs regardless of
+      this option setting.
+    </p>
+
+    <p>
+      To use this option effectively, you should be familiar with reading query profile output
+      to determine the proportion of time spent in the code generation phase, and whether
+      code generation is enabled or not for specific plan fragments.
+    </p>
+
+<!--
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+    </p>
+-->
+
+  </conbody>
+</concept>


[42/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_avro.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_avro.xml b/docs/topics/impala_avro.xml
new file mode 100644
index 0000000..04e84aa
--- /dev/null
+++ b/docs/topics/impala_avro.xml
@@ -0,0 +1,556 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="avro">
+
+  <title>Using the Avro File Format with Impala Tables</title>
+  <titlealts audience="PDF"><navtitle>Avro Data Files</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="File Formats"/>
+      <data name="Category" value="Avro"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="1.4.0">
+      <indexterm audience="Cloudera">Avro support in Impala</indexterm>
+      Impala supports using tables whose data files use the Avro file format. Impala can query Avro
+      tables, and in Impala 1.4.0 and higher can create them, but currently cannot insert data into them. For
+      insert operations, use Hive, then switch back to Impala to run queries.
+    </p>
+
+    <table>
+      <title>Avro Format Support in Impala</title>
+      <tgroup cols="5">
+        <colspec colname="1" colwidth="10*"/>
+        <colspec colname="2" colwidth="10*"/>
+        <colspec colname="3" colwidth="20*"/>
+        <colspec colname="4" colwidth="30*"/>
+        <colspec colname="5" colwidth="30*"/>
+        <thead>
+          <row>
+            <entry>
+              File Type
+            </entry>
+            <entry>
+              Format
+            </entry>
+            <entry>
+              Compression Codecs
+            </entry>
+            <entry>
+              Impala Can CREATE?
+            </entry>
+            <entry>
+              Impala Can INSERT?
+            </entry>
+          </row>
+        </thead>
+        <tbody>
+          <row conref="impala_file_formats.xml#file_formats/avro_support">
+            <entry/>
+          </row>
+        </tbody>
+      </tgroup>
+    </table>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="avro_create_table">
+
+    <title>Creating Avro Tables</title>
+
+    <conbody>
+
+      <p>
+        To create a new table using the Avro file format, issue the <codeph>CREATE TABLE</codeph> statement through
+        Impala with the <codeph>STORED AS AVRO</codeph> clause, or through Hive. If you create the table through
+        Impala, you must include column definitions that match the fields specified in the Avro schema. With Hive,
+        you can omit the columns and just specify the Avro schema.
+      </p>
+
+      <p rev="2.3.0">
+        In <keyword keyref="impala23_full"/> and higher, the <codeph>CREATE TABLE</codeph> for Avro tables can include
+        SQL-style column definitions rather than specifying Avro notation through the <codeph>TBLPROPERTIES</codeph>
+        clause. Impala issues warning messages if there are any mismatches between the types specified in the
+        SQL column definitions and the underlying types; for example, any <codeph>TINYINT</codeph> or
+        <codeph>SMALLINT</codeph> columns are treated as <codeph>INT</codeph> in the underlying Avro files,
+        and therefore are displayed as <codeph>INT</codeph> in any <codeph>DESCRIBE</codeph> or
+        <codeph>SHOW CREATE TABLE</codeph> output.
+      </p>
+
+      <note>
+        <p conref="../shared/impala_common.xml#common/avro_no_timestamp"/>
+      </note>
+
+      <!--
+        To do: Expand these examples to show switching between impala-shell and Hive, loading some data, and then
+        doing DESCRIBE and querying the table.
+      -->
+
+      <p>
+        The following examples demonstrate creating an Avro table in Impala, using either an inline column
+        specification or one taken from a JSON file stored in HDFS:
+      </p>
+
+<codeblock><![CDATA[
+[localhost:21000] > CREATE TABLE avro_only_sql_columns
+                  > (
+                  >   id INT,
+                  >   bool_col BOOLEAN,
+                  >   tinyint_col TINYINT, /* Gets promoted to INT */
+                  >   smallint_col SMALLINT, /* Gets promoted to INT */
+                  >   int_col INT,
+                  >   bigint_col BIGINT,
+                  >   float_col FLOAT,
+                  >   double_col DOUBLE,
+                  >   date_string_col STRING,
+                  >   string_col STRING
+                  > )
+                  > STORED AS AVRO;
+
+[localhost:21000] > CREATE TABLE impala_avro_table
+                  > (bool_col BOOLEAN, int_col INT, long_col BIGINT, float_col FLOAT, double_col DOUBLE, string_col STRING, nullable_int INT)
+                  > STORED AS AVRO
+                  > TBLPROPERTIES ('avro.schema.literal'='{
+                  >    "name": "my_record",
+                  >    "type": "record",
+                  >    "fields": [
+                  >       {"name":"bool_col", "type":"boolean"},
+                  >       {"name":"int_col", "type":"int"},
+                  >       {"name":"long_col", "type":"long"},
+                  >       {"name":"float_col", "type":"float"},
+                  >       {"name":"double_col", "type":"double"},
+                  >       {"name":"string_col", "type":"string"},
+                  >       {"name": "nullable_int", "type": ["null", "int"]}]}');
+
+[localhost:21000] > CREATE TABLE avro_examples_of_all_types (
+                  >     id INT,
+                  >     bool_col BOOLEAN,
+                  >     tinyint_col TINYINT,
+                  >     smallint_col SMALLINT,
+                  >     int_col INT,
+                  >     bigint_col BIGINT,
+                  >     float_col FLOAT,
+                  >     double_col DOUBLE,
+                  >     date_string_col STRING,
+                  >     string_col STRING
+                  >   )
+                  >   STORED AS AVRO
+                  >   TBLPROPERTIES ('avro.schema.url'='hdfs://localhost:8020/avro_schemas/alltypes.json');
+]]>
+</codeblock>
+
+      <p>
+        The following example demonstrates creating an Avro table in Hive:
+      </p>
+
+<codeblock><![CDATA[
+hive> CREATE TABLE hive_avro_table
+    > ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
+    > STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
+    > OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
+    > TBLPROPERTIES ('avro.schema.literal'='{
+    >    "name": "my_record",
+    >    "type": "record",
+    >    "fields": [
+    >       {"name":"bool_col", "type":"boolean"},
+    >       {"name":"int_col", "type":"int"},
+    >       {"name":"long_col", "type":"long"},
+    >       {"name":"float_col", "type":"float"},
+    >       {"name":"double_col", "type":"double"},
+    >       {"name":"string_col", "type":"string"},
+    >       {"name": "nullable_int", "type": ["null", "int"]}]}');
+]]>
+</codeblock>
+
+      <p>
+        Each field of the record becomes a column of the table. Note that any other information, such as the record
+        name, is ignored.
+      </p>
+
+<!-- Have not got a working example of this syntax yet from Lenni.
+<p>
+The schema can be specified either through the <codeph>TBLPROPERTIES</codeph> clause or the
+<codeph>WITH SERDEPROPERTIES</codeph> clause.
+For best compatibility with future versions of Hive, use the <codeph>WITH SERDEPROPERTIES</codeph> clause
+for this information.
+</p>
+-->
+
+      <note>
+        For nullable Avro columns, make sure to put the <codeph>"null"</codeph> entry before the actual type name.
+        In Impala, all columns are nullable; Impala currently does not have a <codeph>NOT NULL</codeph> clause. Any
+        non-nullable property is only enforced on the Avro side.
+      </note>
+
+      <p>
+        Most column types map directly from Avro to Impala under the same names. These are the exceptions and
+        special cases to consider:
+      </p>
+
+      <ul>
+        <li>
+          The <codeph>DECIMAL</codeph> type is defined in Avro as a <codeph>BYTE</codeph> type with the
+          <codeph>logicalType</codeph> property set to <codeph>"decimal"</codeph> and a specified precision and
+          scale. Use <codeph>DECIMAL</codeph> in Avro tables only under CDH 5. The infrastructure and components
+          under CDH 4 do not have reliable <codeph>DECIMAL</codeph> support.
+        </li>
+
+        <li>
+          The Avro <codeph>long</codeph> type maps to <codeph>BIGINT</codeph> in Impala.
+        </li>
+      </ul>
+
+      <p>
+        If you create the table through Hive, switch back to <cmdname>impala-shell</cmdname> and issue an
+        <codeph>INVALIDATE METADATA <varname>table_name</varname></codeph> statement. Then you can run queries for
+        that table through <cmdname>impala-shell</cmdname>.
+      </p>
+
+      <p rev="2.3.0">
+        In rare instances, a mismatch could occur between the Avro schema and the column definitions in the
+        metastore database. In <keyword keyref="impala23_full"/> and higher, Impala checks for such inconsistencies during
+        a <codeph>CREATE TABLE</codeph> statement and each time it loads the metadata for a table (for example,
+        after <codeph>INVALIDATE METADATA</codeph>). Impala uses the following rules to determine how to treat
+        mismatching columns, a process known as <term>schema reconciliation</term>:
+        <ul>
+        <li>
+          If there is a mismatch in the number of columns, Impala uses the column
+          definitions from the Avro schema.
+        </li>
+        <li>
+          If there is a mismatch in column name or type, Impala uses the column definition from the Avro schema.
+          Because a <codeph>CHAR</codeph> or <codeph>VARCHAR</codeph> column in Impala maps to an Avro <codeph>STRING</codeph>,
+          this case is not considered a mismatch and the column is preserved as <codeph>CHAR</codeph> or <codeph>VARCHAR</codeph>
+          in the reconciled schema. <ph rev="2.7.0 IMPALA-3687 CDH-43731">Prior to <keyword keyref="impala27_full"/> the column
+          name and comment for such <codeph>CHAR</codeph> and <codeph>VARCHAR</codeph> columns was also taken from the SQL column definition.
+          In <keyword keyref="impala27_full"/> and higher, the column name and comment from the Avro schema file take precedence for such columns,
+          and only the <codeph>CHAR</codeph> or <codeph>VARCHAR</codeph> type is preserved from the SQL column definition.</ph>
+        </li>
+        <li>
+          An Impala <codeph>TIMESTAMP</codeph> column definition maps to an Avro <codeph>STRING</codeph> and is presented as a <codeph>STRING</codeph>
+          in the reconciled schema, because Avro has no binary <codeph>TIMESTAMP</codeph> representation.
+          As a result, no Avro table can have a <codeph>TIMESTAMP</codeph> column; this restriction is the same as
+          in earlier CDH and Impala releases.
+        </li>
+        </ul>
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_unsupported_filetype"/>
+
+    </conbody>
+  </concept>
+
+  <concept id="avro_map_table">
+
+    <title>Using a Hive-Created Avro Table in Impala</title>
+
+    <conbody>
+
+      <p>
+        If you have an Avro table created through Hive, you can use it in Impala as long as it contains only
+        Impala-compatible data types. It cannot contain:
+        <ul>
+          <li>
+            Complex types: <codeph>array</codeph>, <codeph>map</codeph>, <codeph>record</codeph>,
+            <codeph>struct</codeph>, <codeph>union</codeph> other than
+            <codeph>[<varname>supported_type</varname>,null]</codeph> or
+            <codeph>[null,<varname>supported_type</varname>]</codeph>
+          </li>
+
+          <li>
+            The Avro-specific types <codeph>enum</codeph>, <codeph>bytes</codeph>, and <codeph>fixed</codeph>
+          </li>
+
+          <li>
+            Any scalar type other than those listed in <xref href="impala_datatypes.xml#datatypes"/>
+          </li>
+        </ul>
+        Because Impala and Hive share the same metastore database, Impala can directly access the table definitions
+        and data for tables that were created in Hive.
+      </p>
+
+      <p>
+        If you create an Avro table in Hive, issue an <codeph>INVALIDATE METADATA</codeph> the next time you
+        connect to Impala through <cmdname>impala-shell</cmdname>. This is a one-time operation to make Impala
+        aware of the new table. You can issue the statement while connected to any Impala node, and the catalog
+        service broadcasts the change to all other Impala nodes.
+      </p>
+
+      <p>
+        If you load new data into an Avro table through Hive, either through a Hive <codeph>LOAD DATA</codeph> or
+        <codeph>INSERT</codeph> statement, or by manually copying or moving files into the data directory for the
+        table, issue a <codeph>REFRESH <varname>table_name</varname></codeph> statement the next time you connect
+        to Impala through <cmdname>impala-shell</cmdname>. You can issue the statement while connected to any
+        Impala node, and the catalog service broadcasts the change to all other Impala nodes. If you issue the
+        <codeph>LOAD DATA</codeph> statement through Impala, you do not need a <codeph>REFRESH</codeph> afterward.
+      </p>
+
+      <p>
+        Impala only supports fields of type <codeph>boolean</codeph>, <codeph>int</codeph>, <codeph>long</codeph>,
+        <codeph>float</codeph>, <codeph>double</codeph>, and <codeph>string</codeph>, or unions of these types with
+        null; for example, <codeph>["string", "null"]</codeph>. Unions with <codeph>null</codeph> essentially
+        create a nullable type.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="avro_json">
+
+    <title>Specifying the Avro Schema through JSON</title>
+
+    <conbody>
+
+      <p>
+        While you can embed a schema directly in your <codeph>CREATE TABLE</codeph> statement, as shown above,
+        column width restrictions in the Hive metastore limit the length of schema you can specify. If you
+        encounter problems with long schema literals, try storing your schema as a <codeph>JSON</codeph> file in
+        HDFS instead. Specify your schema in HDFS using table properties similar to the following:
+      </p>
+
+<codeblock>tblproperties ('avro.schema.url'='hdfs//your-name-node:port/path/to/schema.json');</codeblock>
+    </conbody>
+  </concept>
+
+  <concept id="avro_load_data">
+
+    <title>Loading Data into an Avro Table</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="ETL"/>
+      <data name="Category" value="Ingest"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p rev="DOCS-1523">
+        Currently, Impala cannot write Avro data files. Therefore, an Avro table cannot be used as the destination
+        of an Impala <codeph>INSERT</codeph> statement or <codeph>CREATE TABLE AS SELECT</codeph>.
+      </p>
+
+      <p>
+        To copy data from another table, issue any <codeph>INSERT</codeph> statements through Hive. For information
+        about loading data into Avro tables through Hive, see
+        <xref href="https://cwiki.apache.org/confluence/display/Hive/AvroSerDe" scope="external" format="html">Avro
+        page on the Hive wiki</xref>.
+      </p>
+
+      <p>
+        If you already have data files in Avro format, you can also issue <codeph>LOAD DATA</codeph> in either
+        Impala or Hive. Impala can move existing Avro data files into an Avro table, it just cannot create new
+        Avro data files.
+      </p>
+
+    </conbody>
+  </concept>
+
+  <concept id="avro_compression">
+
+    <title>Enabling Compression for Avro Tables</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Compression"/>
+      <data name="Category" value="Snappy"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">compression</indexterm>
+        To enable compression for Avro tables, specify settings in the Hive shell to enable compression and to
+        specify a codec, then issue a <codeph>CREATE TABLE</codeph> statement as in the preceding examples. Impala
+        supports the <codeph>snappy</codeph> and <codeph>deflate</codeph> codecs for Avro tables.
+      </p>
+
+      <p>
+        For example:
+      </p>
+
+<codeblock>hive&gt; set hive.exec.compress.output=true;
+hive&gt; set avro.output.codec=snappy;</codeblock>
+    </conbody>
+  </concept>
+
+  <concept rev="1.1" id="avro_schema_evolution">
+
+    <title>How Impala Handles Avro Schema Evolution</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Concepts"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        Starting in Impala 1.1, Impala can deal with Avro data files that employ <term>schema evolution</term>,
+        where different data files within the same table use slightly different type definitions. (You would
+        perform the schema evolution operation by issuing an <codeph>ALTER TABLE</codeph> statement in the Hive
+        shell.) The old and new types for any changed columns must be compatible, for example a column might start
+        as an <codeph>int</codeph> and later change to a <codeph>bigint</codeph> or <codeph>float</codeph>.
+      </p>
+
+      <p>
+        As with any other tables where the definitions are changed or data is added outside of the current
+        <cmdname>impalad</cmdname> node, ensure that Impala loads the latest metadata for the table if the Avro
+        schema is modified through Hive. Issue a <codeph>REFRESH <varname>table_name</varname></codeph> or
+        <codeph>INVALIDATE METADATA <varname>table_name</varname></codeph> statement. <codeph>REFRESH</codeph>
+        reloads the metadata immediately, <codeph>INVALIDATE METADATA</codeph> reloads the metadata the next time
+        the table is accessed.
+      </p>
+
+      <p>
+        When Avro data files or columns are not consulted during a query, Impala does not check for consistency.
+        Thus, if you issue <codeph>SELECT c1, c2 FROM t1</codeph>, Impala does not return any error if the column
+        <codeph>c3</codeph> changed in an incompatible way. If a query retrieves data from some partitions but not
+        others, Impala does not check the data files for the unused partitions.
+      </p>
+
+      <p>
+        In the Hive DDL statements, you can specify an <codeph>avro.schema.literal</codeph> table property (if the
+        schema definition is short) or an <codeph>avro.schema.url</codeph> property (if the schema definition is
+        long, or to allow convenient editing for the definition).
+      </p>
+
+      <p>
+        For example, running the following SQL code in the Hive shell creates a table using the Avro file format
+        and puts some sample data into it:
+      </p>
+
+<codeblock>CREATE TABLE avro_table (a string, b string)
+ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
+STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
+OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
+TBLPROPERTIES (
+  'avro.schema.literal'='{
+    "type": "record",
+    "name": "my_record",
+    "fields": [
+      {"name": "a", "type": "int"},
+      {"name": "b", "type": "string"}
+    ]}');
+
+INSERT OVERWRITE TABLE avro_table SELECT 1, "avro" FROM functional.alltypes LIMIT 1;
+</codeblock>
+
+      <p>
+        Once the Avro table is created and contains data, you can query it through the
+        <cmdname>impala-shell</cmdname> command:
+      </p>
+
+<codeblock>[localhost:21000] &gt; select * from avro_table;
++---+------+
+| a | b    |
++---+------+
+| 1 | avro |
++---+------+
+</codeblock>
+
+      <p>
+        Now in the Hive shell, you change the type of a column and add a new column with a default value:
+      </p>
+
+<codeblock>-- Promote column "a" from INT to FLOAT (no need to update Avro schema)
+ALTER TABLE avro_table CHANGE A A FLOAT;
+
+-- Add column "c" with default
+ALTER TABLE avro_table ADD COLUMNS (c int);
+ALTER TABLE avro_table SET TBLPROPERTIES (
+  'avro.schema.literal'='{
+    "type": "record",
+    "name": "my_record",
+    "fields": [
+      {"name": "a", "type": "int"},
+      {"name": "b", "type": "string"},
+      {"name": "c", "type": "int", "default": 10}
+    ]}');
+</codeblock>
+
+      <p>
+        Once again in <cmdname>impala-shell</cmdname>, you can query the Avro table based on its latest schema
+        definition. Because the table metadata was changed outside of Impala, you issue a <codeph>REFRESH</codeph>
+        statement first so that Impala has up-to-date metadata for the table.
+      </p>
+
+<codeblock>[localhost:21000] &gt; refresh avro_table;
+[localhost:21000] &gt; select * from avro_table;
++---+------+----+
+| a | b    | c  |
++---+------+----+
+| 1 | avro | 10 |
++---+------+----+
+</codeblock>
+    </conbody>
+  </concept>
+
+  <concept id="avro_data_types">
+
+    <title>Data Type Considerations for Avro Tables</title>
+
+    <conbody>
+
+      <p>
+        The Avro format defines a set of data types whose names differ from the names of the corresponding Impala
+        data types. If you are preparing Avro files using other Hadoop components such as Pig or MapReduce, you
+        might need to work with the type names defined by Avro. The following figure lists the Avro-defined types
+        and the equivalent types in Impala.
+      </p>
+
+<codeblock><![CDATA[Primitive Types (Avro -> Impala)
+--------------------------------
+STRING -> STRING
+STRING -> CHAR
+STRING -> VARCHAR
+INT -> INT
+BOOLEAN -> BOOLEAN
+LONG ->  BIGINT
+FLOAT ->  FLOAT
+DOUBLE -> DOUBLE
+
+Logical Types
+-------------
+BYTES + logicalType = "decimal" -> DECIMAL
+
+Avro Types with No Impala Equivalent
+------------------------------------
+RECORD, MAP, ARRAY, UNION,  ENUM, FIXED, NULL
+
+Impala Types with No Avro Equivalent
+------------------------------------
+TIMESTAMP
+]]>
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/avro_2gb_strings"/>
+
+    </conbody>
+  </concept>
+
+  <concept id="avro_performance">
+
+    <title>Query Performance for Impala Avro Tables</title>
+
+    <conbody>
+
+      <p>
+        In general, expect query performance with Avro tables to be
+        faster than with tables using text data, but slower than with
+        Parquet tables. See <xref href="impala_parquet.xml#parquet"/>
+        for information about using the Parquet file format for
+        high-performance analytic queries.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/s3_block_splitting"/>
+
+    </conbody>
+  </concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_batch_size.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_batch_size.xml b/docs/topics/impala_batch_size.xml
new file mode 100644
index 0000000..eb54b14
--- /dev/null
+++ b/docs/topics/impala_batch_size.xml
@@ -0,0 +1,38 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="batch_size">
+
+  <title>BATCH_SIZE Query Option</title>
+  <titlealts audience="PDF"><navtitle>BATCH_SIZE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Performance"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">BATCH_SIZE query option</indexterm>
+      Number of rows evaluated at a time by SQL operators. Unspecified or a size of 0 uses a predefined default
+      size. Using a large number improves responsiveness, especially for scan operations, at the cost of a higher memory footprint.
+    </p>
+
+    <p>
+      This option is primarily for testing during Impala development, or for use under the direction of <keyword keyref="support_org"/>.
+    </p>
+
+    <p>
+      <b>Type:</b> numeric
+    </p>
+
+    <p>
+      <b>Default:</b> 0 (meaning the predefined default of 1024)
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_bigint.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_bigint.xml b/docs/topics/impala_bigint.xml
new file mode 100644
index 0000000..d9223b0
--- /dev/null
+++ b/docs/topics/impala_bigint.xml
@@ -0,0 +1,102 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="bigint">
+
+  <title>BIGINT Data Type</title>
+  <titlealts audience="PDF"><navtitle>BIGINT</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      An 8-byte integer data type used in <codeph>CREATE TABLE</codeph> and <codeph>ALTER TABLE</codeph>
+      statements.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p>
+      In the column definition of a <codeph>CREATE TABLE</codeph> statement:
+    </p>
+
+<codeblock><varname>column_name</varname> BIGINT</codeblock>
+
+    <p>
+      <b>Range:</b> -9223372036854775808 .. 9223372036854775807. There is no <codeph>UNSIGNED</codeph> subtype.
+    </p>
+
+    <p>
+      <b>Conversions:</b> Impala automatically converts to a floating-point type (<codeph>FLOAT</codeph> or
+      <codeph>DOUBLE</codeph>) automatically. Use <codeph>CAST()</codeph> to convert to <codeph>TINYINT</codeph>,
+      <codeph>SMALLINT</codeph>, <codeph>INT</codeph>, <codeph>STRING</codeph>, or <codeph>TIMESTAMP</codeph>.
+      <ph conref="../shared/impala_common.xml#common/cast_int_to_timestamp"/>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>CREATE TABLE t1 (x BIGINT);
+SELECT CAST(1000 AS BIGINT);
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      <codeph>BIGINT</codeph> is a convenient type to use for column declarations because you can use any kind of
+      integer values in <codeph>INSERT</codeph> statements and they are promoted to <codeph>BIGINT</codeph> where
+      necessary. However, <codeph>BIGINT</codeph> also requires the most bytes of any integer type on disk and in
+      memory, meaning your queries are not as efficient and scalable as possible if you overuse this type.
+      Therefore, prefer to use the smallest integer type with sufficient range to hold all input values, and
+      <codeph>CAST()</codeph> when necessary to the appropriate type.
+    </p>
+
+    <p>
+      For a convenient and automated way to check the bounds of the <codeph>BIGINT</codeph> type, call the
+      functions <codeph>MIN_BIGINT()</codeph> and <codeph>MAX_BIGINT()</codeph>.
+    </p>
+
+    <p>
+      If an integer value is too large to be represented as a <codeph>BIGINT</codeph>, use a
+      <codeph>DECIMAL</codeph> instead with sufficient digits of precision.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/null_bad_numeric_cast"/>
+
+    <p conref="../shared/impala_common.xml#common/partitioning_good"/>
+
+    <p conref="../shared/impala_common.xml#common/hbase_ok"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/parquet_blurb"/> -->
+
+    <p conref="../shared/impala_common.xml#common/text_bulky"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/compatibility_blurb"/> -->
+
+    <p conref="../shared/impala_common.xml#common/internals_8_bytes"/>
+
+    <p conref="../shared/impala_common.xml#common/added_forever"/>
+
+    <p conref="../shared/impala_common.xml#common/column_stats_constant"/>
+
+    <p conref="../shared/impala_common.xml#common/sqoop_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/sqoop_timestamp_caveat"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_literals.xml#numeric_literals"/>, <xref href="impala_tinyint.xml#tinyint"/>,
+      <xref href="impala_smallint.xml#smallint"/>, <xref href="impala_int.xml#int"/>,
+      <xref href="impala_bigint.xml#bigint"/>, <xref href="impala_decimal.xml#decimal"/>,
+      <xref href="impala_math_functions.xml#math_functions"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_bit_functions.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_bit_functions.xml b/docs/topics/impala_bit_functions.xml
new file mode 100644
index 0000000..2ac0f68
--- /dev/null
+++ b/docs/topics/impala_bit_functions.xml
@@ -0,0 +1,794 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="bit_functions" rev="2.3.0">
+
+  <title>Impala Bit Functions</title>
+  <titlealts audience="PDF"><navtitle>Bit Functions</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Querying"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.3.0">
+      Bit manipulation functions perform bitwise operations involved in scientific processing or computer science algorithms.
+      For example, these functions include setting, clearing, or testing bits within an integer value, or changing the
+      positions of bits with or without wraparound.
+    </p>
+
+    <p>
+      If a function takes two integer arguments that are required to be of the same type, the smaller argument is promoted
+      to the type of the larger one if required. For example, <codeph>BITAND(1,4096)</codeph> treats both arguments as
+      <codeph>SMALLINT</codeph>, because 1 can be represented as a <codeph>TINYINT</codeph> but 4096 requires a <codeph>SMALLINT</codeph>.
+    </p>
+
+    <p>
+     Remember that all Impala integer values are signed. Therefore, when dealing with binary values where the most significant
+     bit is 1, the specified or returned values might be negative when represented in base 10.
+    </p>
+
+    <p>
+      Whenever any argument is <codeph>NULL</codeph>, either the input value, bit position, or number of shift or rotate positions,
+      the return value from any of these functions is also <codeph>NULL</codeph>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      The bit functions operate on all the integral data types: <xref href="impala_int.xml#int"/>,
+      <xref href="impala_bigint.xml#bigint"/>, <xref href="impala_smallint.xml#smallint"/>, and
+      <xref href="impala_tinyint.xml#tinyint"/>.
+    </p>
+
+    <p>
+      <b>Function reference:</b>
+    </p>
+
+    <p>
+      Impala supports the following bit functions:
+    </p>
+
+<!--
+bitand
+bitnot
+bitor
+bitxor
+countset
+getbit
+rotateleft
+rotateright
+setbit
+shiftleft
+shiftright
+-->
+
+    <dl>
+
+      <dlentry id="bitand">
+
+        <dt>
+          <codeph>bitand(integer_type a, same_type b)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">bitand() function</indexterm>
+          <b>Purpose:</b> Returns an integer value representing the bits that are set to 1 in both of the arguments.
+          If the arguments are of different sizes, the smaller is promoted to the type of the larger.
+          <p>
+            <b>Usage notes:</b> The <codeph>bitand()</codeph> function is equivalent to the <codeph>&amp;</codeph> binary operator.
+          </p>
+          <p conref="../shared/impala_common.xml#common/return_type_same"/>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples show the results of ANDing integer values.
+            255 contains all 1 bits in its lowermost 7 bits.
+            32767 contains all 1 bits in its lowermost 15 bits.
+            <!--
+            Negative numbers have a 1 in the sign bit and the value is the
+            <xref href="https://en.wikipedia.org/wiki/Two%27s_complement" scope="external" format="html">two's complement</xref>
+            of the positive equivalent.
+            -->
+            You can use the <codeph>bin()</codeph> function to check the binary representation of any
+            integer value, although the result is always represented as a 64-bit value.
+            If necessary, the smaller argument is promoted to the
+            type of the larger one.
+          </p>
+<codeblock>select bitand(255, 32767); /* 0000000011111111 &amp; 0111111111111111 */
++--------------------+
+| bitand(255, 32767) |
++--------------------+
+| 255                |
++--------------------+
+
+select bitand(32767, 1); /* 0111111111111111 &amp; 0000000000000001 */
++------------------+
+| bitand(32767, 1) |
++------------------+
+| 1                |
++------------------+
+
+select bitand(32, 16); /* 00010000 &amp; 00001000 */
++----------------+
+| bitand(32, 16) |
++----------------+
+| 0              |
++----------------+
+
+select bitand(12,5); /* 00001100 &amp; 00000101 */
++---------------+
+| bitand(12, 5) |
++---------------+
+| 4             |
++---------------+
+
+select bitand(-1,15); /* 11111111 &amp; 00001111 */
++----------------+
+| bitand(-1, 15) |
++----------------+
+| 15             |
++----------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="bitnot">
+
+        <dt>
+          <codeph>bitnot(integer_type a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">bitnot() function</indexterm>
+          <b>Purpose:</b> Inverts all the bits of the input argument.
+          <p>
+            <b>Usage notes:</b> The <codeph>bitnot()</codeph> function is equivalent to the <codeph>~</codeph> unary operator.
+          </p>
+          <p conref="../shared/impala_common.xml#common/return_type_same"/>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            These examples illustrate what happens when you flip all the bits of an integer value.
+            The sign always changes. The decimal representation is one different between the positive and
+            negative values.
+            <!--
+            because negative values are represented as the
+            <xref href="https://en.wikipedia.org/wiki/Two%27s_complement" scope="external" format="html">two's complement</xref>
+            of the corresponding positive value.
+            -->
+          </p>
+<codeblock>select bitnot(127); /* 01111111 -> 10000000 */
++-------------+
+| bitnot(127) |
++-------------+
+| -128        |
++-------------+
+
+select bitnot(16); /* 00010000 -> 11101111 */
++------------+
+| bitnot(16) |
++------------+
+| -17        |
++------------+
+
+select bitnot(0); /* 00000000 -> 11111111 */
++-----------+
+| bitnot(0) |
++-----------+
+| -1        |
++-----------+
+
+select bitnot(-128); /* 10000000 -> 01111111 */
++--------------+
+| bitnot(-128) |
++--------------+
+| 127          |
++--------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="bitor">
+
+        <dt>
+          <codeph>bitor(integer_type a, same_type b)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">bitor() function</indexterm>
+          <b>Purpose:</b> Returns an integer value representing the bits that are set to 1 in either of the arguments.
+          If the arguments are of different sizes, the smaller is promoted to the type of the larger.
+          <p>
+            <b>Usage notes:</b> The <codeph>bitor()</codeph> function is equivalent to the <codeph>|</codeph> binary operator.
+          </p>
+          <p conref="../shared/impala_common.xml#common/return_type_same"/>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples show the results of ORing integer values.
+          </p>
+<codeblock>select bitor(1,4); /* 00000001 | 00000100 */
++-------------+
+| bitor(1, 4) |
++-------------+
+| 5           |
++-------------+
+
+select bitor(16,48); /* 00001000 | 00011000 */
++---------------+
+| bitor(16, 48) |
++---------------+
+| 48            |
++---------------+
+
+select bitor(0,7); /* 00000000 | 00000111 */
++-------------+
+| bitor(0, 7) |
++-------------+
+| 7           |
++-------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="bitxor">
+
+        <dt>
+          <codeph>bitxor(integer_type a, same_type b)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">bitxor() function</indexterm>
+          <b>Purpose:</b> Returns an integer value representing the bits that are set to 1 in one but not both of the arguments.
+          If the arguments are of different sizes, the smaller is promoted to the type of the larger.
+          <p>
+            <b>Usage notes:</b> The <codeph>bitxor()</codeph> function is equivalent to the <codeph>^</codeph> binary operator.
+          </p>
+          <p conref="../shared/impala_common.xml#common/return_type_same"/>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples show the results of XORing integer values.
+            XORing a non-zero value with zero returns the non-zero value.
+            XORing two identical values returns zero, because all the 1 bits from the first argument are also 1 bits in the second argument.
+            XORing different non-zero values turns off some bits and leaves others turned on, based on whether the same bit is set in both arguments.
+          </p>
+<codeblock>select bitxor(0,15); /* 00000000 ^ 00001111 */
++---------------+
+| bitxor(0, 15) |
++---------------+
+| 15            |
++---------------+
+
+select bitxor(7,7); /* 00000111 ^ 00000111 */
++--------------+
+| bitxor(7, 7) |
++--------------+
+| 0            |
++--------------+
+
+select bitxor(8,4); /* 00001000 ^ 00000100 */
++--------------+
+| bitxor(8, 4) |
++--------------+
+| 12           |
++--------------+
+
+select bitxor(3,7); /* 00000011 ^ 00000111 */
++--------------+
+| bitxor(3, 7) |
++--------------+
+| 4            |
++--------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="countset">
+
+        <dt>
+          <codeph>countset(integer_type a [, int zero_or_one])</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">countset() function</indexterm>
+          <b>Purpose:</b> By default, returns the number of 1 bits in the specified integer value.
+          If the optional second argument is set to zero, it returns the number of 0 bits instead.
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            In discussions of information theory, this operation is referred to as the
+            <q><xref href="https://en.wikipedia.org/wiki/Hamming_weight" scope="external" format="html">population count</xref></q>
+            or <q>popcount</q>.
+          </p>
+          <p conref="../shared/impala_common.xml#common/return_type_same"/>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples show how to count the number of 1 bits in an integer value.
+          </p>
+<codeblock>select countset(1); /* 00000001 */
++-------------+
+| countset(1) |
++-------------+
+| 1           |
++-------------+
+
+select countset(3); /* 00000011 */
++-------------+
+| countset(3) |
++-------------+
+| 2           |
++-------------+
+
+select countset(16); /* 00010000 */
++--------------+
+| countset(16) |
++--------------+
+| 1            |
++--------------+
+
+select countset(17); /* 00010001 */
++--------------+
+| countset(17) |
++--------------+
+| 2            |
++--------------+
+
+select countset(7,1); /* 00000111 = 3 1 bits; the function counts 1 bits by default */
++----------------+
+| countset(7, 1) |
++----------------+
+| 3              |
++----------------+
+
+select countset(7,0); /* 00000111 = 5 0 bits; third argument can only be 0 or 1 */
++----------------+
+| countset(7, 0) |
++----------------+
+| 5              |
++----------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="getbit">
+
+        <dt>
+          <codeph>getbit(integer_type a, int position)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">getbit() function</indexterm>
+          <b>Purpose:</b> Returns a 0 or 1 representing the bit at a
+          specified position. The positions are numbered right to left, starting at zero.
+          The position argument cannot be negative.
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            When you use a literal input value, it is treated as an 8-bit, 16-bit,
+            and so on value, the smallest type that is appropriate.
+            The type of the input value limits the range of the positions.
+            Cast the input value to the appropriate type if you need to
+            ensure it is treated as a 64-bit, 32-bit, and so on value.
+          </p>
+          <p conref="../shared/impala_common.xml#common/return_type_same"/>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples show how to test a specific bit within an integer value.
+          </p>
+<codeblock>select getbit(1,0); /* 00000001 */
++--------------+
+| getbit(1, 0) |
++--------------+
+| 1            |
++--------------+
+
+select getbit(16,1) /* 00010000 */
++---------------+
+| getbit(16, 1) |
++---------------+
+| 0             |
++---------------+
+
+select getbit(16,4) /* 00010000 */
++---------------+
+| getbit(16, 4) |
++---------------+
+| 1             |
++---------------+
+
+select getbit(16,5) /* 00010000 */
++---------------+
+| getbit(16, 5) |
++---------------+
+| 0             |
++---------------+
+
+select getbit(-1,3); /* 11111111 */
++---------------+
+| getbit(-1, 3) |
++---------------+
+| 1             |
++---------------+
+
+select getbit(-1,25); /* 11111111 */
+ERROR: Invalid bit position: 25
+
+select getbit(cast(-1 as int),25); /* 11111111111111111111111111111111 */
++-----------------------------+
+| getbit(cast(-1 as int), 25) |
++-----------------------------+
+| 1                           |
++-----------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="rotateleft">
+
+        <dt>
+          <codeph>rotateleft(integer_type a, int positions)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">rotateleft() function</indexterm>
+          <b>Purpose:</b> Rotates an integer value left by a specified number of bits.
+          As the most significant bit is taken out of the original value,
+          if it is a 1 bit, it is <q>rotated</q> back to the least significant bit.
+          Therefore, the final value has the same number of 1 bits as the original value,
+          just in different positions.
+          In computer science terms, this operation is a
+          <q><xref href="https://en.wikipedia.org/wiki/Circular_shift" scope="external" format="html">circular shift</xref></q>.
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            Specifying a second argument of zero leaves the original value unchanged.
+            Rotating a -1 value by any number of positions still returns -1,
+            because the original value has all 1 bits and all the 1 bits are
+            preserved during rotation.
+            Similarly, rotating a 0 value by any number of positions still returns 0.
+            Rotating a value by the same number of bits as in the value returns the same value.
+            Because this is a circular operation, the number of positions is not limited
+            to the number of bits in the input value.
+            For example, rotating an 8-bit value by 1, 9, 17, and so on positions returns an
+            identical result in each case.
+          </p>
+          <p conref="../shared/impala_common.xml#common/return_type_same"/>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>select rotateleft(1,4); /* 00000001 -> 00010000 */
++------------------+
+| rotateleft(1, 4) |
++------------------+
+| 16               |
++------------------+
+
+select rotateleft(-1,155); /* 11111111 -> 11111111 */
++---------------------+
+| rotateleft(-1, 155) |
++---------------------+
+| -1                  |
++---------------------+
+
+select rotateleft(-128,1); /* 10000000 -> 00000001 */
++---------------------+
+| rotateleft(-128, 1) |
++---------------------+
+| 1                   |
++---------------------+
+
+select rotateleft(-127,3); /* 10000001 -> 00001100 */
++---------------------+
+| rotateleft(-127, 3) |
++---------------------+
+| 12                  |
++---------------------+
+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="rotateright">
+
+        <dt>
+          <codeph>rotateright(integer_type a, int positions)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">rotateright() function</indexterm>
+          <b>Purpose:</b> Rotates an integer value right by a specified number of bits.
+          As the least significant bit is taken out of the original value,
+          if it is a 1 bit, it is <q>rotated</q> back to the most significant bit.
+          Therefore, the final value has the same number of 1 bits as the original value,
+          just in different positions.
+          In computer science terms, this operation is a
+          <q><xref href="https://en.wikipedia.org/wiki/Circular_shift" scope="external" format="html">circular shift</xref></q>.
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            Specifying a second argument of zero leaves the original value unchanged.
+            Rotating a -1 value by any number of positions still returns -1,
+            because the original value has all 1 bits and all the 1 bits are
+            preserved during rotation.
+            Similarly, rotating a 0 value by any number of positions still returns 0.
+            Rotating a value by the same number of bits as in the value returns the same value.
+            Because this is a circular operation, the number of positions is not limited
+            to the number of bits in the input value.
+            For example, rotating an 8-bit value by 1, 9, 17, and so on positions returns an
+            identical result in each case.
+          </p>
+          <p conref="../shared/impala_common.xml#common/return_type_same"/>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>select rotateright(16,4); /* 00010000 -> 00000001 */
++--------------------+
+| rotateright(16, 4) |
++--------------------+
+| 1                  |
++--------------------+
+
+select rotateright(-1,155); /* 11111111 -> 11111111 */
++----------------------+
+| rotateright(-1, 155) |
++----------------------+
+| -1                   |
++----------------------+
+
+select rotateright(-128,1); /* 10000000 -> 01000000 */
++----------------------+
+| rotateright(-128, 1) |
++----------------------+
+| 64                   |
++----------------------+
+
+select rotateright(-127,3); /* 10000001 -> 00110000 */
++----------------------+
+| rotateright(-127, 3) |
++----------------------+
+| 48                   |
++----------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="setbit">
+
+        <dt>
+          <codeph>setbit(integer_type a, int position [, int zero_or_one])</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">setbit() function</indexterm>
+          <b>Purpose:</b> By default, changes a bit at a specified position to a 1, if it is not already.
+          If the optional third argument is set to zero, the specified bit is set to 0 instead.
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          If the bit at the specified position was already 1 (by default)
+          or 0 (with a third argument of zero), the return value is
+          the same as the first argument.
+          The positions are numbered right to left, starting at zero.
+          (Therefore, the return value could be different from the first argument
+          even if the position argument is zero.)
+          The position argument cannot be negative.
+          <p>
+            When you use a literal input value, it is treated as an 8-bit, 16-bit,
+            and so on value, the smallest type that is appropriate.
+            The type of the input value limits the range of the positions.
+            Cast the input value to the appropriate type if you need to
+            ensure it is treated as a 64-bit, 32-bit, and so on value.
+          </p>
+          <p conref="../shared/impala_common.xml#common/return_type_same"/>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>select setbit(0,0); /* 00000000 -> 00000001 */
++--------------+
+| setbit(0, 0) |
++--------------+
+| 1            |
++--------------+
+
+select setbit(0,3); /* 00000000 -> 00001000 */
++--------------+
+| setbit(0, 3) |
++--------------+
+| 8            |
++--------------+
+
+select setbit(7,3); /* 00000111 -> 00001111 */
++--------------+
+| setbit(7, 3) |
++--------------+
+| 15           |
++--------------+
+
+select setbit(15,3); /* 00001111 -> 00001111 */
++---------------+
+| setbit(15, 3) |
++---------------+
+| 15            |
++---------------+
+
+select setbit(0,32); /* By default, 0 is a TINYINT with only 8 bits. */
+ERROR: Invalid bit position: 32
+
+select setbit(cast(0 as bigint),32); /* For BIGINT, the position can be 0..63. */
++-------------------------------+
+| setbit(cast(0 as bigint), 32) |
++-------------------------------+
+| 4294967296                    |
++-------------------------------+
+
+select setbit(7,3,1); /* 00000111 -> 00001111; setting to 1 is the default */
++-----------------+
+| setbit(7, 3, 1) |
++-----------------+
+| 15              |
++-----------------+
+
+select setbit(7,2,0); /* 00000111 -> 00000011; third argument of 0 clears instead of sets */
++-----------------+
+| setbit(7, 2, 0) |
++-----------------+
+| 3               |
++-----------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="shiftleft">
+
+        <dt>
+          <codeph>shiftleft(integer_type a, int positions)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">shiftleft() function</indexterm>
+          <b>Purpose:</b> Shifts an integer value left by a specified number of bits.
+          As the most significant bit is taken out of the original value,
+          it is discarded and the least significant bit becomes 0.
+          In computer science terms, this operation is a <q><xref href="https://en.wikipedia.org/wiki/Logical_shift" scope="external" format="html">logical shift</xref></q>.
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            The final value has either the same number of 1 bits as the original value, or fewer.
+            Shifting an 8-bit value by 8 positions, a 16-bit value by 16 positions, and so on produces
+            a result of zero.
+          </p>
+          <p>
+            Specifying a second argument of zero leaves the original value unchanged.
+            Shifting any value by 0 returns the original value.
+            Shifting any value by 1 is the same as multiplying it by 2,
+            as long as the value is small enough; larger values eventually
+            become negative when shifted, as the sign bit is set.
+            Starting with the value 1 and shifting it left by N positions gives
+            the same result as 2 to the Nth power, or <codeph>pow(2,<varname>N</varname>)</codeph>.
+          </p>
+          <p conref="../shared/impala_common.xml#common/return_type_same"/>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>select shiftleft(1,0); /* 00000001 -> 00000001 */
++-----------------+
+| shiftleft(1, 0) |
++-----------------+
+| 1               |
++-----------------+
+
+select shiftleft(1,3); /* 00000001 -> 00001000 */
++-----------------+
+| shiftleft(1, 3) |
++-----------------+
+| 8               |
++-----------------+
+
+select shiftleft(8,2); /* 00001000 -> 00100000 */
++-----------------+
+| shiftleft(8, 2) |
++-----------------+
+| 32              |
++-----------------+
+
+select shiftleft(127,1); /* 01111111 -> 11111110 */
++-------------------+
+| shiftleft(127, 1) |
++-------------------+
+| -2                |
++-------------------+
+
+select shiftleft(127,5); /* 01111111 -> 11100000 */
++-------------------+
+| shiftleft(127, 5) |
++-------------------+
+| -32               |
++-------------------+
+
+select shiftleft(-1,4); /* 11111111 -> 11110000 */
++------------------+
+| shiftleft(-1, 4) |
++------------------+
+| -16              |
++------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="shiftright">
+
+        <dt>
+          <codeph>shiftright(integer_type a, int positions)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">shiftright() function</indexterm>
+          <b>Purpose:</b> Shifts an integer value right by a specified number of bits.
+          As the least significant bit is taken out of the original value,
+          it is discarded and the most significant bit becomes 0.
+          In computer science terms, this operation is a <q><xref href="https://en.wikipedia.org/wiki/Logical_shift" scope="external" format="html">logical shift</xref></q>.
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+          Therefore, the final value has either the same number of 1 bits as the original value, or fewer.
+          Shifting an 8-bit value by 8 positions, a 16-bit value by 16 positions, and so on produces
+          a result of zero.
+          </p>
+          <p>
+            Specifying a second argument of zero leaves the original value unchanged.
+            Shifting any value by 0 returns the original value.
+            Shifting any positive value right by 1 is the same as dividing it by 2.
+            Negative values become positive when shifted right.
+          </p>
+          <p conref="../shared/impala_common.xml#common/return_type_same"/>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>select shiftright(16,0); /* 00010000 -> 00000000 */
++-------------------+
+| shiftright(16, 0) |
++-------------------+
+| 16                |
++-------------------+
+
+select shiftright(16,4); /* 00010000 -> 00000000 */
++-------------------+
+| shiftright(16, 4) |
++-------------------+
+| 1                 |
++-------------------+
+
+select shiftright(16,5); /* 00010000 -> 00000000 */
++-------------------+
+| shiftright(16, 5) |
++-------------------+
+| 0                 |
++-------------------+
+
+select shiftright(-1,1); /* 11111111 -> 01111111 */
++-------------------+
+| shiftright(-1, 1) |
++-------------------+
+| 127               |
++-------------------+
+
+select shiftright(-1,5); /* 11111111 -> 00000111 */
++-------------------+
+| shiftright(-1, 5) |
++-------------------+
+| 7                 |
++-------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+    </dl>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_boolean.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_boolean.xml b/docs/topics/impala_boolean.xml
new file mode 100644
index 0000000..c788b33
--- /dev/null
+++ b/docs/topics/impala_boolean.xml
@@ -0,0 +1,154 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="boolean">
+
+  <title>BOOLEAN Data Type</title>
+  <titlealts audience="PDF"><navtitle>BOOLEAN</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      A data type used in <codeph>CREATE TABLE</codeph> and <codeph>ALTER TABLE</codeph> statements, representing a
+      single true/false choice.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p>
+      In the column definition of a <codeph>CREATE TABLE</codeph> statement:
+    </p>
+
+<codeblock><varname>column_name</varname> BOOLEAN</codeblock>
+
+    <p>
+      <b>Range:</b> <codeph>TRUE</codeph> or <codeph>FALSE</codeph>. Do not use quotation marks around the
+      <codeph>TRUE</codeph> and <codeph>FALSE</codeph> literal values. You can write the literal values in
+      uppercase, lowercase, or mixed case. The values queried from a table are always returned in lowercase,
+      <codeph>true</codeph> or <codeph>false</codeph>.
+    </p>
+
+    <p>
+      <b>Conversions:</b> Impala does not automatically convert any other type to <codeph>BOOLEAN</codeph>. All
+      conversions must use an explicit call to the <codeph>CAST()</codeph> function.
+    </p>
+
+    <p>
+      You can use <codeph>CAST()</codeph> to convert
+<!--
+      <codeph>TINYINT</codeph>, <codeph>SMALLINT</codeph>,
+      <codeph>INT</codeph>, <codeph>BIGINT</codeph>, <codeph>FLOAT</codeph>, or <codeph>DOUBLE</codeph>
+-->
+      any integer or floating-point type to
+      <codeph>BOOLEAN</codeph>: a value of 0 represents <codeph>false</codeph>, and any non-zero value is converted
+      to <codeph>true</codeph>.
+    </p>
+
+<codeblock>SELECT CAST(42 AS BOOLEAN) AS nonzero_int, CAST(99.44 AS BOOLEAN) AS nonzero_decimal,
+  CAST(000 AS BOOLEAN) AS zero_int, CAST(0.0 AS BOOLEAN) AS zero_decimal;
++-------------+-----------------+----------+--------------+
+| nonzero_int | nonzero_decimal | zero_int | zero_decimal |
++-------------+-----------------+----------+--------------+
+| true        | true            | false    | false        |
++-------------+-----------------+----------+--------------+
+</codeblock>
+
+    <p>
+      When you cast the opposite way, from <codeph>BOOLEAN</codeph> to a numeric type,
+      the result becomes either 1 or 0:
+    </p>
+
+<codeblock>SELECT CAST(true AS INT) AS true_int, CAST(true AS DOUBLE) AS true_double,
+  CAST(false AS INT) AS false_int, CAST(false AS DOUBLE) AS false_double;
++----------+-------------+-----------+--------------+
+| true_int | true_double | false_int | false_double |
++----------+-------------+-----------+--------------+
+| 1        | 1           | 0         | 0            |
++----------+-------------+-----------+--------------+
+</codeblock>
+
+    <p rev="1.4.0">
+<!-- BOOLEAN-to-DECIMAL casting requested in IMPALA-991. As of Sept. 2014, designated "won't fix". -->
+      You can cast <codeph>DECIMAL</codeph> values to <codeph>BOOLEAN</codeph>, with the same treatment of zero and
+      non-zero values as the other numeric types. You cannot cast a <codeph>BOOLEAN</codeph> to a
+      <codeph>DECIMAL</codeph>.
+    </p>
+
+    <p>
+      You cannot cast a <codeph>STRING</codeph> value to <codeph>BOOLEAN</codeph>, although you can cast a
+      <codeph>BOOLEAN</codeph> value to <codeph>STRING</codeph>, returning <codeph>'1'</codeph> for
+      <codeph>true</codeph> values and <codeph>'0'</codeph> for <codeph>false</codeph> values.
+    </p>
+
+    <p>
+      Although you can cast a <codeph>TIMESTAMP</codeph> to a <codeph>BOOLEAN</codeph> or a
+      <codeph>BOOLEAN</codeph> to a <codeph>TIMESTAMP</codeph>, the results are unlikely to be useful. Any non-zero
+      <codeph>TIMESTAMP</codeph> (that is, any value other than <codeph>1970-01-01 00:00:00</codeph>) becomes
+      <codeph>TRUE</codeph> when converted to <codeph>BOOLEAN</codeph>, while <codeph>1970-01-01 00:00:00</codeph>
+      becomes <codeph>FALSE</codeph>. A value of <codeph>FALSE</codeph> becomes <codeph>1970-01-01
+      00:00:00</codeph> when converted to <codeph>BOOLEAN</codeph>, and <codeph>TRUE</codeph> becomes one second
+      past this epoch date, that is, <codeph>1970-01-01 00:00:01</codeph>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/null_null_arguments"/>
+
+    <p conref="../shared/impala_common.xml#common/partitioning_blurb"/>
+
+    <p>
+      Do not use a <codeph>BOOLEAN</codeph> column as a partition key. Although you can create such a table,
+      subsequent operations produce errors:
+    </p>
+
+<codeblock>[localhost:21000] &gt; create table truth_table (assertion string) partitioned by (truth boolean);
+[localhost:21000] &gt; insert into truth_table values ('Pigs can fly',false);
+ERROR: AnalysisException: INSERT into table with BOOLEAN partition column (truth) is not supported: partitioning.truth_table
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>SELECT 1 &lt; 2;
+SELECT 2 = 5;
+SELECT 100 &lt; NULL, 100 &gt; NULL;
+CREATE TABLE assertions (claim STRING, really BOOLEAN);
+INSERT INTO assertions VALUES
+  ("1 is less than 2", 1 &lt; 2),
+  ("2 is the same as 5", 2 = 5),
+  ("Grass is green", true),
+  ("The moon is made of green cheese", false);
+SELECT claim FROM assertions WHERE really = TRUE;
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/hbase_ok"/>
+
+    <p conref="../shared/impala_common.xml#common/parquet_ok"/>
+
+    <p conref="../shared/impala_common.xml#common/text_bulky"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/compatibility_blurb"/> -->
+
+<!-- <p conref="../shared/impala_common.xml#common/internals_blurb"/> -->
+
+<!-- <p conref="../shared/impala_common.xml#common/added_in_20"/> -->
+
+    <p conref="../shared/impala_common.xml#common/column_stats_constant"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/restrictions_blurb"/> -->
+
+<!-- <p conref="../shared/impala_common.xml#common/related_info"/> -->
+
+    <p>
+      <b>Related information:</b> <xref href="impala_literals.xml#boolean_literals"/>,
+      <xref href="impala_operators.xml#operators"/>,
+      <xref href="impala_conditional_functions.xml#conditional_functions"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_breakpad.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_breakpad.xml b/docs/topics/impala_breakpad.xml
new file mode 100644
index 0000000..f764c2f
--- /dev/null
+++ b/docs/topics/impala_breakpad.xml
@@ -0,0 +1,256 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="breakpad" rev="2.6.0 IMPALA-2686 CDH-40238">
+
+  <title>Breakpad Minidumps for Impala (<keyword keyref="impala26"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>Breakpad Minidumps</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Support"/>
+      <data name="Category" value="Administrators"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.6.0 IMPALA-2686 CDH-40238">
+      The <xref href="https://chromium.googlesource.com/breakpad/breakpad/" scope="external" format="html">breakpad</xref>
+      project is an open-source framework for crash reporting.
+      In <keyword keyref="impala26_full"/> and higher, Impala can use <codeph>breakpad</codeph> to record stack information and
+      register values when any of the Impala-related daemons crash due to an error such as <codeph>SIGSEGV</codeph>
+      or unhandled exceptions.
+      The dump files are much smaller than traditional core dump files. The dump mechanism itself uses very little
+      memory, which improves reliability if the crash occurs while the system is low on memory.
+    </p>
+
+    <note type="important">
+      Because of the internal mechanisms involving Impala memory allocation and Linux
+      signalling for out-of-memory (OOM) errors, if an Impala-related daemon experiences a
+      crash due to an OOM condition, it does <i>not</i> generate a minidump for that error.
+    <p>
+
+    </p>
+    </note>
+
+
+    <p outputclass="toc inpage" audience="PDF"/>
+
+  </conbody>
+
+  <concept id="breakpad_minidump_enable">
+    <title>Enabling or Disabling Minidump Generation</title>
+    <conbody>
+      <p>
+        By default, a minidump file is generated when an Impala-related daemon crashes.
+        To turn off generation of the minidump files, change the
+        <uicontrol>minidump_path</uicontrol> configuration setting of one or more Impala-related daemons
+        to the empty string, and restart the corresponding services or daemons.
+      </p>
+
+      <p rev="IMPALA-3677 CDH-43745">
+        In <keyword keyref="impala27_full"/> and higher,
+        you can send a <codeph>SIGUSR1</codeph> signal to any Impala-related daemon to write a
+        Breakpad minidump. For advanced troubleshooting, you can now produce a minidump
+        without triggering a crash.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="breakpad_minidump_location" rev="IMPALA-3581">
+    <title>Specifying the Location for Minidump Files</title>
+    <conbody>
+      <p>
+        By default, all minidump files are written to the following location
+        on the host where a crash occurs:
+        <!-- Location stated in IMPALA-3581; overridden by different location from IMPALA-2686?
+             <filepath><varname>log_directory</varname>/minidumps/<varname>daemon_name</varname></filepath> -->
+         <ul>
+          <li>
+            <p>
+              Clusters managed by Cloudera Manager: <filepath>/var/log/impala-minidumps/<varname>daemon_name</varname></filepath>
+            </p>
+          </li>
+          <li>
+            <p>
+              Clusters not managed by Cloudera Manager:
+              <filepath><varname>impala_log_dir</varname>/<varname>daemon_name</varname>/minidumps/<varname>daemon_name</varname></filepath>
+            </p>
+          </li>
+        </ul>
+        The minidump files for <cmdname>impalad</cmdname>, <cmdname>catalogd</cmdname>,
+        and <cmdname>statestored</cmdname> are each written to a separate directory.
+      </p>
+      <p>
+        To specify a different location, set the
+        <!-- Again, IMPALA-3581 says one thing and IMPALA-2686 / observation of CM interface says another.
+             <codeph>log_dir</codeph> -->
+        <uicontrol>minidump_path</uicontrol>
+        configuration setting of one or more Impala-related daemons, and restart the corresponding services or daemons.
+      </p>
+      <p>
+        If you specify a relative path for this setting, the value is interpreted relative to
+        the default <uicontrol>minidump_path</uicontrol> directory.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="breakpad_minidump_number">
+    <title>Controlling the Number of Minidump Files</title>
+    <conbody>
+      <p>
+        Like any files used for logging or troubleshooting, consider limiting the number of
+        minidump files, or removing unneeded ones, depending on the amount of free storage
+        space on the hosts in the cluster.
+      </p>
+      <p>
+        Because the minidump files are only used for problem resolution, you can remove any such files that
+        are not needed to debug current issues.
+      </p>
+      <p>
+        To control how many minidump files Impala keeps around at any one time,
+        set the <uicontrol>max_minidumps</uicontrol> configuration setting for
+        of one or more Impala-related daemon, and restart the corresponding services or daemons.
+        The default for this setting is 9. A zero or negative value is interpreted as
+        <q>unlimited</q>.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="breakpad_minidump_logging">
+    <title>Detecting Crash Events</title>
+    <conbody>
+      <p>
+        You can see in the Impala log files or in the Cloudera Manager charts for Impala
+        when crash events occur that generate minidump files. Because each restart begins
+        a new log file, the <q>crashed</q> message is always at or near the bottom of the
+        log file. (There might be another later message if core dumps are also enabled.)
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="breakpad_support_process" rev="CDH-39818">
+    <title>Using the Minidump Files for Problem Resolution</title>
+    <conbody>
+      <p>
+        Typically, you provide minidump files to <keyword keyref="support_org"/> as part of problem resolution,
+        in the same way that you might provide a core dump. The <uicontrol>Send Diagnostic Data</uicontrol>
+        under the <uicontrol>Support</uicontrol> menu in Cloudera Manager guides you through the
+        process of selecting a time period and volume of diagnostic data, then collects the data
+        from all hosts and transmits the relevant information for you.
+      </p>
+      <fig id="fig_pqw_gvx_pr">
+        <title>Send Diagnostic Data choice under Support menu</title>
+        <image href="../images/support_send_diagnostic_data.png" scalefit="yes" placement="break"/>
+      </fig>
+      <p>
+        You might get additional instructions from <keyword keyref="support_org"/> about collecting minidumps to better isolate a specific problem.
+        Because the information in the minidump files is limited to stack traces and register contents,
+        the possibility of including sensitive information is much lower than with core dump files.
+        If any sensitive information is included in the minidump, <keyword keyref="support_org"/> preserves the confidentiality of that information.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="breakpad_demo">
+    <title>Demonstration of Breakpad Feature</title>
+    <conbody>
+      <p>
+        The following example uses the command <cmdname>kill -11</cmdname> to
+        simulate a <codeph>SIGSEGV</codeph> crash for an <cmdname>impalad</cmdname>
+        process on a single DataNode, then examines the relevant log files and minidump file.
+      </p>
+      <p>
+        First, as root on a worker node, we kill the <cmdname>impalad</cmdname> process with a
+        <codeph>SIGSEGV</codeph> error. The original process ID was 23114. (Cloudera Manager
+        restarts the process with a new pid, as shown by the second <cmdname>ps</cmdname> command.)
+      </p>
+<codeblock><![CDATA[
+# ps ax | grep impalad
+23114 ?        Sl     0:18 /opt/cloudera/parcels/<parcel_version>/lib/impala/sbin-retail/impalad --flagfile=/var/run/cloudera-scm-agent/process/114-impala-IMPALAD/impala-conf/impalad_flags
+31259 pts/0    S+     0:00 grep impalad
+# 
+# kill -11 23114
+#
+# ps ax | grep impalad
+31374 ?        Rl     0:04 /opt/cloudera/parcels/<parcel_version>/lib/impala/sbin-retail/impalad --flagfile=/var/run/cloudera-scm-agent/process/114-impala-IMPALAD/impala-conf/impalad_flags
+31475 pts/0    S+     0:00 grep impalad
+]]>
+</codeblock>
+
+      <p>
+        We locate the log directory underneath <filepath>/var/log</filepath>.
+        There is a <codeph>.INFO</codeph>, <codeph>.WARNING</codeph>, and <codeph>.ERROR</codeph>
+        log file for the 23114 process ID. The minidump message is written to the
+        <codeph>.INFO</codeph> file and the <codeph>.ERROR</codeph> file, but not the
+        <codeph>.WARNING</codeph> file. In this case, a large core file was also produced.
+      </p>
+<codeblock><![CDATA[
+# cd /var/log/impalad
+# ls -la | grep 23114
+-rw-------   1 impala impala 3539079168 Jun 23 15:20 core.23114
+-rw-r--r--   1 impala impala      99057 Jun 23 15:20 hs_err_pid23114.log
+-rw-r--r--   1 impala impala        351 Jun 23 15:20 impalad.worker_node_123.impala.log.ERROR.20160623-140343.23114
+-rw-r--r--   1 impala impala      29101 Jun 23 15:20 impalad.worker_node_123.impala.log.INFO.20160623-140343.23114
+-rw-r--r--   1 impala impala        228 Jun 23 14:03 impalad.worker_node_123.impala.log.WARNING.20160623-140343.23114
+]]>
+</codeblock>
+      <p>
+        The <codeph>.INFO</codeph> log includes the location of the minidump file, followed by
+        a report of a core dump. With the breakpad minidump feature enabled, now we might
+        disable core dumps or keep fewer of them around.
+      </p>
+<codeblock><![CDATA[
+# cat impalad.worker_node_123.impala.log.INFO.20160623-140343.23114
+...
+Wrote minidump to /var/log/impala-minidumps/impalad/0980da2d-a905-01e1-25ff883a-04ee027a.dmp
+#
+# A fatal error has been detected by the Java Runtime Environment:
+#
+#  SIGSEGV (0xb) at pc=0x00000030c0e0b68a, pid=23114, tid=139869541455968
+#
+# JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
+# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
+# Problematic frame:
+# C  [libpthread.so.0+0xb68a]  pthread_cond_wait+0xca
+#
+# Core dump written. Default location: /var/log/impalad/core or core.23114
+#
+# An error report file with more information is saved as:
+# /var/log/impalad/hs_err_pid23114.log
+#
+# If you would like to submit a bug report, please visit:
+#   http://bugreport.sun.com/bugreport/crash.jsp
+# The crash happened outside the Java Virtual Machine in native code.
+# See problematic frame for where to report the bug.
+...
+
+# cat impalad.worker_node_123.impala.log.ERROR.20160623-140343.23114
+
+Log file created at: 2016/06/23 14:03:43
+Running on machine:.worker_node_123
+Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
+E0623 14:03:43.911002 23114 logging.cc:118] stderr will be logged to this file.
+Wrote minidump to /var/log/impala-minidumps/impalad/0980da2d-a905-01e1-25ff883a-04ee027a.dmp
+]]>
+</codeblock>
+      <p>
+        The resulting minidump file is much smaller than the corresponding core file,
+        making it much easier to supply diagnostic information to <keyword keyref="support_org"/>.
+        The transmission process for the minidump files is automated through Cloudera Manager.
+      </p>
+<codeblock><![CDATA[
+# pwd
+/var/log/impalad
+# cd ../impala-minidumps/impalad
+# ls
+0980da2d-a905-01e1-25ff883a-04ee027a.dmp
+# du -kh *
+2.4M  0980da2d-a905-01e1-25ff883a-04ee027a.dmp
+]]>
+</codeblock>
+    </conbody>
+  </concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_cdh.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_cdh.xml b/docs/topics/impala_cdh.xml
new file mode 100644
index 0000000..7ecba74
--- /dev/null
+++ b/docs/topics/impala_cdh.xml
@@ -0,0 +1,25 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="impala_cdh">
+
+  <title>How Impala Works with CDH</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Concepts"/>
+      <data name="Category" value="CDH"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p conref="../shared/impala_common.xml#common/impala_overview_diagram"/>
+
+    <p conref="../shared/impala_common.xml#common/component_list"/>
+
+    <p conref="../shared/impala_common.xml#common/query_overview"/>
+  </conbody>
+</concept>


[06/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_show.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_show.xml b/docs/topics/impala_show.xml
new file mode 100644
index 0000000..fbee88a
--- /dev/null
+++ b/docs/topics/impala_show.xml
@@ -0,0 +1,1264 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="show">
+
+  <title>SHOW Statement</title>
+  <titlealts audience="PDF"><navtitle>SHOW</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Reports"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">SHOW statement</indexterm>
+      The <codeph>SHOW</codeph> statement is a flexible way to get information about different types of Impala
+      objects.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>SHOW DATABASES [[LIKE] '<varname>pattern</varname>']
+SHOW SCHEMAS [[LIKE] '<varname>pattern</varname>'] - an alias for SHOW DATABASES
+SHOW TABLES [IN <varname>database_name</varname>] [[LIKE] '<varname>pattern</varname>']
+<ph rev="1.2.0">SHOW [AGGREGATE | ANALYTIC] FUNCTIONS [IN <varname>database_name</varname>] [[LIKE] '<varname>pattern</varname>']</ph>
+<ph rev="1.2.1">SHOW CREATE TABLE [<varname>database_name</varname>].<varname>table_name</varname></ph>
+<ph rev="1.2.1">SHOW TABLE STATS [<varname>database_name</varname>.]<varname>table_name</varname></ph>
+<ph rev="1.2.1">SHOW COLUMN STATS [<varname>database_name</varname>.]<varname>table_name</varname></ph>
+<ph rev="1.4.0">SHOW PARTITIONS [<varname>database_name</varname>.]<varname>table_name</varname></ph>
+<ph rev="2.2.0">SHOW FILES IN [<varname>database_name</varname>.]<varname>table_name</varname> [PARTITION (<varname>key_col</varname>=<varname>value</varname> [, <varname>key_col</varname>=<varname>value</varname>]]</ph>
+
+<ph rev="2.0.0">SHOW ROLES
+SHOW CURRENT ROLES
+SHOW ROLE GRANT GROUP <varname>group_name</varname>
+SHOW GRANT ROLE <varname>role_name</varname></ph>
+</codeblock>
+
+<!-- SHOW ROLE GRANT { USER <varname>user_name</varname> | GROUP <varname>group_name</varname> | ROLE <varname>role_name</varname> } -->
+
+<!-- Extracted from the previous codeblock because even hidden content produces blank lines.
+<ph audience="Cloudera" rev="1.4.0">SHOW DATA SOURCES [LIKE '<varname>source_name</varname>]</ph>
+-->
+
+<!-- Some suggestion there would be this syntax for 1.4, but it's not in the builds:
+<ph rev="1.4.0">SHOW [CACHED] TABLES [IN <varname>database_name</varname>] [[LIKE] '<varname>pattern</varname>']</ph>
+<ph rev="1.4.0">SHOW [CACHED] PARTITIONS [<varname>database_name</varname>.]<varname>table_name</varname></ph>
+-->
+
+    <p>
+      Issue a <codeph>SHOW <varname>object_type</varname></codeph> statement to see the appropriate objects in the
+      current database, or <codeph>SHOW <varname>object_type</varname> IN <varname>database_name</varname></codeph>
+      to see objects in a specific database.
+    </p>
+
+    <p>
+      The optional <varname>pattern</varname> argument is a quoted string literal, using Unix-style
+      <codeph>*</codeph> wildcards and allowing <codeph>|</codeph> for alternation. The preceding
+      <codeph>LIKE</codeph> keyword is also optional. All object names are stored in lowercase, so use all
+      lowercase letters in the pattern string. For example:
+    </p>
+
+<codeblock>show databases 'a*';
+show databases like 'a*';
+show tables in some_db like '*fact*';
+use some_db;
+show tables '*dim*|*fact*';</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p outputclass="toc inpage"/>
+
+  </conbody>
+
+  <concept rev="2.2.0" id="show_files">
+
+    <title>SHOW FILES Statement</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Disk Storage"/>
+      <data name="Category" value="Tables"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        The <codeph>SHOW FILES</codeph> statement displays the files that constitute a specified table,
+        or a partition within a partitioned table. This syntax is available in <keyword keyref="impala22_full"/> and higher
+        only. The output includes the names of the files, the size of each file, and the applicable partition
+        for a partitioned table. The size includes a suffix of <codeph>B</codeph> for bytes,
+        <codeph>MB</codeph> for megabytes, and <codeph>GB</codeph> for gigabytes.
+      </p>
+
+      <note>
+        This statement applies to tables and partitions stored on HDFS, or in the Amazon Simple Storage System (S3).
+        It does not apply to views.
+        It does not apply to tables mapped onto HBase, because HBase does not use the same file-based storage layout.
+      </note>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p>
+        You can use this statement to verify the results of your ETL process: that is, that
+        the expected files are present, with the expected sizes. You can examine the file information
+        to detect conditions such as empty files, missing files, or inefficient layouts due to
+        a large number of small files. When you use <codeph>INSERT</codeph> statements to copy
+        from one table to another, you can see how the file layout changes due to file format
+        conversions, compaction of small input files into large data blocks, and
+        multiple output files from parallel queries and partitioned inserts.
+      </p>
+
+      <p>
+        The output from this statement does not include files that Impala considers to be hidden
+        or invisible, such as those whose names start with a dot or an underscore, or that
+        end with the suffixes <codeph>.copying</codeph> or <codeph>.tmp</codeph>.
+      </p>
+
+      <p>
+        The information for partitioned tables complements the output of the <codeph>SHOW PARTITIONS</codeph>
+        statement, which summarizes information about each partition. <codeph>SHOW PARTITIONS</codeph>
+        produces some output for each partition, while <codeph>SHOW FILES</codeph> does not
+        produce any output for empty partitions because they do not include any data files.
+      </p>
+
+<!-- Extensive round of testing makes me pretty confident of these findings. -->
+      <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+      <p rev="CDH-19187">
+        The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+        typically the <codeph>impala</codeph> user, must have read
+        permission for all the table files, read and execute permission for all the directories that make up the table,
+        and execute permission for the database directory and all its parent directories.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following example shows a <codeph>SHOW FILES</codeph> statement
+        for an unpartitioned table using text format:
+      </p>
+
+<codeblock scale="60">[localhost:21000] > create table unpart_text (x bigint, s string);
+[localhost:21000] > insert into unpart_text (x, s) select id, name
+                  > from oreilly.sample_data limit 20e6;
+[localhost:21000] > show files in unpart_text;
++------------------------------------------------------------------------------+----------+-----------+
+| path                                                                         | size     | partition |
++------------------------------------------------------------------------------+----------+-----------+
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/unpart_text/35665776ef85cfaf_1012432410_data.0. | 448.31MB |           |
++------------------------------------------------------------------------------+----------+-----------+
+[localhost:21000] > insert into unpart_text (x, s) select id, name from oreilly.sample_data limit 100e6;
+[localhost:21000] > show files in unpart_text;
++--------------------------------------------------------------------------------------+----------+-----------+
+| path                                                                                 | size     | partition |
++--------------------------------------------------------------------------------------+----------+-----------+
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/unpart_text/35665776ef85cfaf_1012432410_data.0. | 448.31MB |           |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/unpart_text/ac3dba252a8952b8_1663177415_data.0. | 2.19GB   |           |
++--------------------------------------------------------------------------------------+----------+-----------+
+</codeblock>
+
+      <p>
+        This example illustrates how, after issuing some <codeph>INSERT ... VALUES</codeph> statements,
+        the table now contains some tiny files of just a few bytes. Such small files could cause inefficient processing of
+        parallel queries that are expecting multi-megabyte input files. The example shows how you might compact the small files by doing
+        an <codeph>INSERT ... SELECT</codeph> into a different table, possibly converting the data to Parquet in the process:
+      </p>
+
+<codeblock scale="60">[localhost:21000] > insert into unpart_text values (10,'hello'), (20, 'world');
+[localhost:21000] > insert into unpart_text values (-1,'foo'), (-1000, 'bar');
+[localhost:21000] > show files in unpart_text;
++--------------------------------------------------------------------------------------+----------+
+| path                                                                                 | size     |
++--------------------------------------------------------------------------------------+----------+
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/unpart_text/4f11b8bdf8b6aa92_238145083_data.0.  | 18B
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/unpart_text/35665776ef85cfaf_1012432410_data.0. | 448.31MB
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/unpart_text/ac3dba252a8952b8_1663177415_data.0. | 2.19GB
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/unpart_text/cfb8252452445682_1868457216_data.0. | 17B
++--------------------------------------------------------------------------------------+----------+
+[localhost:21000] > create table unpart_parq stored as parquet as select * from unpart_text;
++---------------------------+
+| summary                   |
++---------------------------+
+| Inserted 120000002 row(s) |
++---------------------------+
+[localhost:21000] > show files in unpart_parq;
++------------------------------------------------------------------------------------------+----------+
+| path                                                                                     | size     |
++------------------------------------------------------------------------------------------+----------+
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/unpart_parq/60798d96ba630184_549959007_data.0.parq  | 255.36MB |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/unpart_parq/60798d96ba630184_549959007_data.1.parq  | 178.52MB |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/unpart_parq/60798d96ba630185_549959007_data.0.parq  | 255.37MB |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/unpart_parq/60798d96ba630185_549959007_data.1.parq  | 57.71MB  |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/unpart_parq/60798d96ba630186_2141167244_data.0.parq | 255.40MB |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/unpart_parq/60798d96ba630186_2141167244_data.1.parq | 175.52MB |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/unpart_parq/60798d96ba630187_1006832086_data.0.parq | 255.40MB |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/unpart_parq/60798d96ba630187_1006832086_data.1.parq | 214.61MB |
++------------------------------------------------------------------------------------------+----------+
+</codeblock>
+
+      <p>
+        The following example shows a <codeph>SHOW FILES</codeph> statement for a partitioned text table
+        with data in two different partitions, and two empty partitions.
+        The partitions with no data are not represented in the <codeph>SHOW FILES</codeph> output.
+      </p>
+<codeblock scale="60">[localhost:21000] > create table part_text (x bigint, y int, s string)
+                                        > partitioned by (year bigint, month bigint, day bigint);
+[localhost:21000] > insert overwrite part_text (x, y, s) partition (year=2014,month=1,day=1)
+                  > select id, val, name from oreilly.normalized_parquet
+where id between 1 and 1000000;
+[localhost:21000] > insert overwrite part_text (x, y, s) partition (year=2014,month=1,day=2)
+                  > select id, val, name from oreilly.normalized_parquet
+                  > where id between 1000001 and 2000000;
+[localhost:21000] > alter table part_text add partition (year=2014,month=1,day=3);
+[localhost:21000] > alter table part_text add partition (year=2014,month=1,day=4);
+[localhost:21000] > show partitions part_text;
++-------+-------+-----+-------+--------+---------+--------------+-------------------+--------+-------------------+
+| year  | month | day | #Rows | #Files | Size    | Bytes Cached | Cache Replication | Format | Incremental stats |
++-------+-------+-----+-------+--------+---------+--------------+-------------------+--------+-------------------+
+| 2014  | 1     | 1   | -1    | 4      | 25.16MB | NOT CACHED   | NOT CACHED        | TEXT   | false             |
+| 2014  | 1     | 2   | -1    | 4      | 26.22MB | NOT CACHED   | NOT CACHED        | TEXT   | false             |
+| 2014  | 1     | 3   | -1    | 0      | 0B      | NOT CACHED   | NOT CACHED        | TEXT   | false             |
+| 2014  | 1     | 4   | -1    | 0      | 0B      | NOT CACHED   | NOT CACHED        | TEXT   | false             |
+| Total |       |     | -1    | 8      | 51.38MB | 0B           |                   |        |                   |
++-------+-------+-----+-------+--------+---------+--------------+-------------------+--------+-------------------+
+[localhost:21000] > show files in part_text;
++---------------------------------------------------------------------------------------------------------+--------+-------------------------+
+| path                                                                                                    | size   | partition               |
++---------------------------------------------------------------------------------------------------------+--------+-------------------------+
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/part_text/year=2014/month=1/day=1/80732d9dc80689f_1418645991_data.0.  | 5.77MB | year=2014/month=1/day=1 |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/part_text/year=2014/month=1/day=1/80732d9dc8068a0_1418645991_data.0.  | 6.25MB | year=2014/month=1/day=1 |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/part_text/year=2014/month=1/day=1/80732d9dc8068a1_147082319_data.0.   | 7.16MB | year=2014/month=1/day=1 |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/part_text/year=2014/month=1/day=1/80732d9dc8068a2_2111411753_data.0.  | 5.98MB | year=2014/month=1/day=1 |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/part_text/year=2014/month=1/day=2/21a828cf494b5bbb_501271652_data.0.  | 6.42MB | year=2014/month=1/day=2 |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/part_text/year=2014/month=1/day=2/21a828cf494b5bbc_501271652_data.0.  | 6.62MB | year=2014/month=1/day=2 |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/part_text/year=2014/month=1/day=2/21a828cf494b5bbd_1393490200_data.0. | 6.98MB | year=2014/month=1/day=2 |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/part_text/year=2014/month=1/day=2/21a828cf494b5bbe_1393490200_data.0. | 6.20MB | year=2014/month=1/day=2 |
++---------------------------------------------------------------------------------------------------------+--------+-------------------------+
+</codeblock>
+      <p>
+        The following example shows a <codeph>SHOW FILES</codeph> statement for a partitioned Parquet table.
+        The number and sizes of files are different from the equivalent partitioned text table
+        used in the previous example, because <codeph>INSERT</codeph> operations for Parquet tables
+        are parallelized differently than for text tables. (Also, the amount of data is so small
+        that it can be written to Parquet without involving all the hosts in this 4-node cluster.)
+      </p>
+<codeblock scale="60">[localhost:21000] > create table part_parq (x bigint, y int, s string) partitioned by (year bigint, month bigint, day bigint) stored as parquet;
+[localhost:21000] > insert into part_parq partition (year,month,day) select x, y, s, year, month, day from partitioned_text;
+[localhost:21000] > show partitions part_parq;
++-------+-------+-----+-------+--------+---------+--------------+-------------------+---------+-------------------+
+| year  | month | day | #Rows | #Files | Size    | Bytes Cached | Cache Replication | Format  | Incremental stats |
++-------+-------+-----+-------+--------+---------+--------------+-------------------+---------+-------------------+
+| 2014  | 1     | 1   | -1    | 3      | 17.89MB | NOT CACHED   | NOT CACHED        | PARQUET | false             |
+| 2014  | 1     | 2   | -1    | 3      | 17.89MB | NOT CACHED   | NOT CACHED        | PARQUET | false             |
+| Total |       |     | -1    | 6      | 35.79MB | 0B           |                   |         |                   |
++-------+-------+-----+-------+--------+---------+--------------+-------------------+---------+-------------------+
+[localhost:21000] > show files in part_parq;
++-----------------------------------------------------------------------------------------------+--------+-------------------------+
+| path                                                                                          | size   | partition               |
++-----------------------------------------------------------------------------------------------+--------+-------------------------+
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/part_parq/year=2014/month=1/day=1/1134113650_data.0.parq | 4.49MB | year=2014/month=1/day=1 |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/part_parq/year=2014/month=1/day=1/617567880_data.0.parq  | 5.14MB | year=2014/month=1/day=1 |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/part_parq/year=2014/month=1/day=1/2099499416_data.0.parq | 8.27MB | year=2014/month=1/day=1 |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/part_parq/year=2014/month=1/day=2/945567189_data.0.parq  | 8.80MB | year=2014/month=1/day=2 |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/part_parq/year=2014/month=1/day=2/2145850112_data.0.parq | 4.80MB | year=2014/month=1/day=2 |
+| hdfs://<varname>impala_data_dir</varname>/show_files.db/part_parq/year=2014/month=1/day=2/665613448_data.0.parq  | 4.29MB | year=2014/month=1/day=2 |
++-----------------------------------------------------------------------------------------------+--------+-------------------------+
+</codeblock>
+<p>
+  The following example shows output from the <codeph>SHOW FILES</codeph> statement
+  for a table where the data files are stored in Amazon S3:
+</p>
+<codeblock scale="70">[localhost:21000] > show files in s3_testing.sample_data_s3;
++-----------------------------------------------------------------------+---------+
+| path                                                                  | size    |
++-----------------------------------------------------------------------+---------+
+| s3a://impala-demo/sample_data/e065453cba1988a6_1733868553_data.0.parq | 24.84MB |
++-----------------------------------------------------------------------+---------+
+</codeblock>
+<!--
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_authorization.xml#authorization"/>
+      </p>
+-->
+    </conbody>
+  </concept>
+
+  <concept rev="2.0.0" id="show_roles">
+
+    <title>SHOW ROLES Statement</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Sentry"/>
+      <data name="Category" value="Roles"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        The <codeph>SHOW ROLES</codeph> statement displays roles. This syntax is available in CDH 5.2 and later
+        only, when you are using the Sentry authorization framework along with the Sentry service, as described in
+        <xref href="impala_authorization.xml#sentry_service"/>. It does not apply when you use the Sentry framework
+        with privileges defined in a policy file.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/security_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/show_security"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        Depending on the roles set up within your organization by the <codeph>CREATE ROLE</codeph> statement, the
+        output might look something like this:
+      </p>
+
+<codeblock>show roles;
++-----------+
+| role_name |
++-----------+
+| analyst   |
+| role1     |
+| sales     |
+| superuser |
+| test_role |
++-----------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/permissions_blurb_no"/>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_authorization.xml#authorization"/>
+      </p>
+    </conbody>
+  </concept>
+
+  <concept rev="2.0.0" id="show_current_role">
+
+    <title>SHOW CURRENT ROLE</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Sentry"/>
+      <data name="Category" value="Roles"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p rev="2.0.0">
+        The <codeph>SHOW CURRENT ROLE</codeph> statement displays roles assigned to the current user. This syntax
+        is available in CDH 5.2 and later only, when you are using the Sentry authorization framework along with
+        the Sentry service, as described in <xref href="impala_authorization.xml#sentry_service"/>. It does not
+        apply when you use the Sentry framework with privileges defined in a policy file.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/security_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/show_security"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        Depending on the roles set up within your organization by the <codeph>CREATE ROLE</codeph> statement, the
+        output might look something like this:
+      </p>
+
+<codeblock>show current roles;
++-----------+
+| role_name |
++-----------+
+| role1     |
+| superuser |
++-----------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/permissions_blurb_no"/>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_authorization.xml#authorization"/>
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="show_role_grant">
+
+    <title>SHOW ROLE GRANT Statement</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Sentry"/>
+      <data name="Category" value="Roles"/>
+    </metadata>
+  </prolog>
+
+
+    <conbody>
+
+      <p rev="2.0.0">
+<!-- Copied from Sentry docs. Turn into conref. I did some rewording for clarity. -->
+        The <codeph>SHOW ROLE GRANT</codeph> statement lists all the roles assigned to the specified group. This
+        statement is only allowed for Sentry administrative users and others users that are part of the specified
+        group. This syntax is available in CDH 5.2 and later only, when you are using the Sentry authorization
+        framework along with the Sentry service, as described in
+        <xref href="impala_authorization.xml#sentry_service"/>. It does not apply when you use the Sentry framework
+        with privileges defined in a policy file.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/security_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/show_security"/>
+
+<!--
+<p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>To do: construct example for SHOW ROLE GRANT</codeblock>
+-->
+
+      <p conref="../shared/impala_common.xml#common/permissions_blurb_no"/>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_authorization.xml#authorization"/>
+      </p>
+    </conbody>
+  </concept>
+
+  <concept rev="2.0.0" id="show_grant_role">
+
+    <title>SHOW GRANT ROLE Statement</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Sentry"/>
+      <data name="Category" value="Roles"/>
+    </metadata>
+  </prolog>
+
+
+    <conbody>
+
+      <p>
+<!-- Copied from Sentry docs. Turn into conref. I did some rewording for clarity. -->
+        The <codeph>SHOW GRANT ROLE</codeph> statement list all the grants for the given role name. This statement
+        is only allowed for Sentry administrative users and other users that have been granted the specified role.
+        This syntax is available in CDH 5.2 and later only, when you are using the Sentry authorization framework
+        along with the Sentry service, as described in <xref href="impala_authorization.xml#sentry_service"/>. It
+        does not apply when you use the Sentry framework with privileges defined in a policy file.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/security_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/show_security"/>
+
+<!--
+<p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>To do: construct example for SHOW GRANT ROLE</codeblock>
+-->
+
+      <p conref="../shared/impala_common.xml#common/permissions_blurb_no"/>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_authorization.xml#authorization"/>
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="show_databases">
+
+    <title>SHOW DATABASES</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Databases"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        The <codeph>SHOW DATABASES</codeph> statement is often the first one you issue when connecting to an
+        instance for the first time. You typically issue <codeph>SHOW DATABASES</codeph> to see the names you can
+        specify in a <codeph>USE <varname>db_name</varname></codeph> statement, then after switching to a database
+        you issue <codeph>SHOW TABLES</codeph> to see the names you can specify in <codeph>SELECT</codeph> and
+        <codeph>INSERT</codeph> statements.
+      </p>
+
+      <p rev="2.5.0 IMPALA-2070">
+        In <keyword keyref="impala25_full"/> and higher, the output includes a second column showing any associated comment
+        for each database.
+      </p>
+
+      <p>
+        The output of <codeph>SHOW DATABASES</codeph> includes the special <codeph>_impala_builtins</codeph>
+        database, which lets you view definitions of built-in functions, as described under <codeph>SHOW
+        FUNCTIONS</codeph>.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/security_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/show_security"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        This example shows how you might locate a particular table on an unfamiliar system. The
+        <codeph>DEFAULT</codeph> database is the one you initially connect to; a database with that name is present
+        on every system. You can issue <codeph>SHOW TABLES IN <varname>db_name</varname></codeph> without going
+        into a database, or <codeph>SHOW TABLES</codeph> once you are inside a particular database.
+      </p>
+
+<codeblock rev="2.5.0 IMPALA-2070">[localhost:21000] &gt; show databases;
++------------------+----------------------------------------------+
+| name             | comment                                      |
++------------------+----------------------------------------------+
+| _impala_builtins | System database for Impala builtin functions |
+| default          | Default Hive database                        |
+| file_formats     |                                              |
++------------------+----------------------------------------------+
+Returned 3 row(s) in 0.02s
+[localhost:21000] &gt; show tables in file_formats;
++--------------------+
+| name               |
++--------------------+
+| parquet_table      |
+| rcfile_table       |
+| sequencefile_table |
+| textfile_table     |
++--------------------+
+Returned 4 row(s) in 0.01s
+[localhost:21000] &gt; use file_formats;
+[localhost:21000] &gt; show tables like '*parq*';
++--------------------+
+| name               |
++--------------------+
+| parquet_table      |
++--------------------+
+Returned 1 row(s) in 0.01s</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/permissions_blurb_no"/>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_databases.xml#databases"/>, <xref href="impala_create_database.xml#create_database"/>,
+        <xref href="impala_drop_database.xml#drop_database"/>, <xref href="impala_use.xml#use"/>
+        <xref href="impala_show.xml#show_tables"/>,
+        <xref href="impala_show.xml#show_functions"/>
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="show_tables">
+
+    <title>SHOW TABLES Statement</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        Displays the names of tables. By default, lists tables in the current database, or with the
+        <codeph>IN</codeph> clause, in a specified database. By default, lists all tables, or with the
+        <codeph>LIKE</codeph> clause, only those whose name match a pattern with <codeph>*</codeph> wildcards.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/security_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/show_security"/>
+
+      <p rev="CDH-19187">
+        The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+        typically the <codeph>impala</codeph> user, must have read and execute
+        permissions for all directories that are part of the table.
+        (A table could span multiple different HDFS directories if it is partitioned.
+        The directories could be widely scattered because a partition can reside
+        in an arbitrary HDFS directory based on its <codeph>LOCATION</codeph> attribute.)
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following examples demonstrate the <codeph>SHOW TABLES</codeph> statement.
+        If the database contains no tables, the result set is empty.
+        If the database does contain tables, <codeph>SHOW TABLES IN <varname>db_name</varname></codeph>
+        lists all the table names. <codeph>SHOW TABLES</codeph> with no qualifiers lists
+        all the table names in the current database.
+      </p>
+
+<codeblock>create database empty_db;
+show tables in empty_db;
+Fetched 0 row(s) in 0.11s
+
+create database full_db;
+create table full_db.t1 (x int);
+create table full_db.t2 like full_db.t1;
+
+show tables in full_db;
++------+
+| name |
++------+
+| t1   |
+| t2   |
++------+
+
+use full_db;
+show tables;
++------+
+| name |
++------+
+| t1   |
+| t2   |
++------+
+</codeblock>
+
+      <p>
+        This example demonstrates how <codeph>SHOW TABLES LIKE '<varname>wildcard_pattern</varname>'</codeph>
+        lists table names that match a pattern, or multiple alternative patterns.
+        The ability to do wildcard matches for table names makes it helpful to establish naming conventions for tables to
+        conveniently locate a group of related tables.
+      </p>
+
+<codeblock>create table fact_tbl (x int);
+create table dim_tbl_1 (s string);
+create table dim_tbl_2 (s string);
+
+/* Asterisk is the wildcard character. Only 2 out of the 3 just-created tables are returned. */
+show tables like 'dim*';
++-----------+
+| name      |
++-----------+
+| dim_tbl_1 |
+| dim_tbl_2 |
++-----------+
+
+/* We are already in the FULL_DB database, but just to be sure we can specify the database name also. */
+show tables in full_db like 'dim*';
++-----------+
+| name      |
++-----------+
+| dim_tbl_1 |
+| dim_tbl_2 |
++-----------+
+
+/* The pipe character separates multiple wildcard patterns. */
+show tables like '*dim*|t*';
++-----------+
+| name      |
++-----------+
+| dim_tbl_1 |
+| dim_tbl_2 |
+| t1        |
+| t2        |
++-----------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/permissions_blurb_no"/>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_tables.xml#tables"/>, <xref href="impala_create_table.xml#create_table"/>,
+        <xref href="impala_alter_table.xml#alter_table"/>, <xref href="impala_drop_table.xml#drop_table"/>,
+        <xref href="impala_describe.xml#describe"/>, <xref href="impala_show.xml#show_create_table"/>,
+        <xref href="impala_show.xml#show_table_stats"/>,
+        <xref href="impala_show.xml#show_databases"/>,
+        <xref href="impala_show.xml#show_functions"/>
+      </p>
+    </conbody>
+  </concept>
+
+  <concept rev="1.2.1" id="show_create_table">
+
+    <title>SHOW CREATE TABLE Statement</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Schemas"/>
+      <data name="Category" value="Impala Data Types"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        As a schema changes over time, you might run a <codeph>CREATE TABLE</codeph> statement followed by several
+        <codeph>ALTER TABLE</codeph> statements. To capture the cumulative effect of all those statements,
+        <codeph>SHOW CREATE TABLE</codeph> displays a <codeph>CREATE TABLE</codeph> statement that would reproduce
+        the current structure of a table. You can use this output in scripts that set up or clone a group of
+        tables, rather than trying to reproduce the original sequence of <codeph>CREATE TABLE</codeph> and
+        <codeph>ALTER TABLE</codeph> statements. When creating variations on the original table, or cloning the
+        original table on a different system, you might need to edit the <codeph>SHOW CREATE TABLE</codeph> output
+        to change things such as the database name, <codeph>LOCATION</codeph> field, and so on that might be
+        different on the destination system.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/security_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/show_security"/>
+
+      <p conref="../shared/impala_common.xml#common/permissions_blurb_no"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following example shows how various clauses from the <codeph>CREATE TABLE</codeph> statement are
+        represented in the output of <codeph>SHOW CREATE TABLE</codeph>.
+      </p>
+
+<codeblock>create table show_create_table_demo (id int comment "Unique ID", y double, s string)
+  partitioned by (year smallint)
+  stored as parquet;
+
+show create table show_create_table_demo;
++----------------------------------------------------------------------------------------+
+| result                                                                                 |
++----------------------------------------------------------------------------------------+
+| CREATE TABLE scratch.show_create_table_demo (                                          |
+|   id INT COMMENT 'Unique ID',                                                          |
+|   y DOUBLE,                                                                            |
+|   s STRING                                                                             |
+| )                                                                                      |
+| PARTITIONED BY (                                                                       |
+|   year SMALLINT                                                                        |
+| )                                                                                      |
+| STORED AS PARQUET                                                                      |
+| LOCATION 'hdfs://127.0.0.1:8020/user/hive/warehouse/scratch.db/show_create_table_demo' |
+| TBLPROPERTIES ('transient_lastDdlTime'='1418152582')                                   |
++----------------------------------------------------------------------------------------+
+</codeblock>
+
+      <p>
+        The following example shows how, after a sequence of <codeph>ALTER TABLE</codeph> statements, the output
+        from <codeph>SHOW CREATE TABLE</codeph> represents the current state of the table. This output could be
+        used to create a matching table rather than executing the original <codeph>CREATE TABLE</codeph> and
+        sequence of <codeph>ALTER TABLE</codeph> statements.
+      </p>
+
+<codeblock>alter table show_create_table_demo drop column s;
+alter table show_create_table_demo set fileformat textfile;
+
+show create table show_create_table_demo;
++----------------------------------------------------------------------------------------+
+| result                                                                                 |
++----------------------------------------------------------------------------------------+
+| CREATE TABLE scratch.show_create_table_demo (                                          |
+|   id INT COMMENT 'Unique ID',                                                          |
+|   y DOUBLE                                                                             |
+| )                                                                                      |
+| PARTITIONED BY (                                                                       |
+|   year SMALLINT                                                                        |
+| )                                                                                      |
+| STORED AS TEXTFILE                                                                     |
+| LOCATION 'hdfs://127.0.0.1:8020/user/hive/warehouse/demo.db/show_create_table_demo'    |
+| TBLPROPERTIES ('transient_lastDdlTime'='1418152638')                                   |
++----------------------------------------------------------------------------------------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_create_table.xml#create_table"/>, <xref href="impala_describe.xml#describe"/>,
+        <xref href="impala_show.xml#show_tables"/>
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="show_table_stats">
+
+    <title>SHOW TABLE STATS Statement</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Performance"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        The <codeph>SHOW TABLE STATS</codeph> and <codeph>SHOW COLUMN STATS</codeph> variants are important for
+        tuning performance and diagnosing performance issues, especially with the largest tables and the most
+        complex join queries.
+      </p>
+
+      <p>
+        Any values that are not available (because the <codeph>COMPUTE STATS</codeph> statement has not been run
+        yet) are displayed as <codeph>-1</codeph>.
+      </p>
+
+      <p>
+        <codeph>SHOW TABLE STATS</codeph> provides some general information about the table, such as the number of
+        files, overall size of the data, whether some or all of the data is in the HDFS cache, and the file format,
+        that is useful whether or not you have run the <codeph>COMPUTE STATS</codeph> statement. A
+        <codeph>-1</codeph> in the <codeph>#Rows</codeph> output column indicates that the <codeph>COMPUTE
+        STATS</codeph> statement has never been run for this table. If the table is partitioned, <codeph>SHOW TABLE
+        STATS</codeph> provides this information for each partition. (It produces the same output as the
+        <codeph>SHOW PARTITIONS</codeph> statement in this case.)
+      </p>
+
+      <p>
+        The output of <codeph>SHOW COLUMN STATS</codeph> is primarily only useful after the <codeph>COMPUTE
+        STATS</codeph> statement has been run on the table. A <codeph>-1</codeph> in the <codeph>#Distinct
+        Values</codeph> output column indicates that the <codeph>COMPUTE STATS</codeph> statement has never been
+        run for this table. Currently, Impala always leaves the <codeph>#Nulls</codeph> column as
+        <codeph>-1</codeph>, even after <codeph>COMPUTE STATS</codeph> has been run.
+      </p>
+
+      <p>
+        These <codeph>SHOW</codeph> statements work on actual tables only, not on views.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/security_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/show_security"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following examples show how the <codeph>SHOW TABLE STATS</codeph> statement displays physical
+        information about a table and the associated data files:
+      </p>
+
+<codeblock>show table stats store_sales;
++-------+--------+----------+--------------+--------+-------------------+
+| #Rows | #Files | Size     | Bytes Cached | Format | Incremental stats |
++-------+--------+----------+--------------+--------+-------------------+
+| -1    | 1      | 370.45MB | NOT CACHED   | TEXT   | false             |
++-------+--------+----------+--------------+--------+-------------------+
+
+show table stats customer;
++-------+--------+---------+--------------+--------+-------------------+
+| #Rows | #Files | Size    | Bytes Cached | Format | Incremental stats |
++-------+--------+---------+--------------+--------+-------------------+
+| -1    | 1      | 12.60MB | NOT CACHED   | TEXT   | false             |
++-------+--------+---------+--------------+--------+-------------------+
+</codeblock>
+
+      <p>
+        The following example shows how, after a <codeph>COMPUTE STATS</codeph> or <codeph>COMPUTE INCREMENTAL
+        STATS</codeph> statement, the <codeph>#Rows</codeph> field is now filled in. Because the
+        <codeph>STORE_SALES</codeph> table in this example is not partitioned, the <codeph>COMPUTE INCREMENTAL
+        STATS</codeph> statement produces regular stats rather than incremental stats, therefore the
+        <codeph>Incremental stats</codeph> field remains <codeph>false</codeph>.
+      </p>
+
+<codeblock>compute stats customer;
++------------------------------------------+
+| summary                                  |
++------------------------------------------+
+| Updated 1 partition(s) and 18 column(s). |
++------------------------------------------+
+
+show table stats customer;
++--------+--------+---------+--------------+--------+-------------------+
+| #Rows  | #Files | Size    | Bytes Cached | Format | Incremental stats |
++--------+--------+---------+--------------+--------+-------------------+
+| 100000 | 1      | 12.60MB | NOT CACHED   | TEXT   | false             |
++--------+--------+---------+--------------+--------+-------------------+
+
+compute incremental stats store_sales;
++------------------------------------------+
+| summary                                  |
++------------------------------------------+
+| Updated 1 partition(s) and 23 column(s). |
++------------------------------------------+
+
+show table stats store_sales;
++---------+--------+----------+--------------+--------+-------------------+
+| #Rows   | #Files | Size     | Bytes Cached | Format | Incremental stats |
++---------+--------+----------+--------------+--------+-------------------+
+| 2880404 | 1      | 370.45MB | NOT CACHED   | TEXT   | false             |
++---------+--------+----------+--------------+--------+-------------------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+      <p rev="CDH-19187">
+        The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+        typically the <codeph>impala</codeph> user, must have read and execute
+        permissions for all directories that are part of the table.
+        (A table could span multiple different HDFS directories if it is partitioned.
+        The directories could be widely scattered because a partition can reside
+        in an arbitrary HDFS directory based on its <codeph>LOCATION</codeph> attribute.)
+        The Impala user must also have execute
+        permission for the database directory, and any parent directories of the database directory in HDFS.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_compute_stats.xml#compute_stats"/>, <xref href="impala_show.xml#show_column_stats"/>
+      </p>
+
+      <p>
+        See <xref href="impala_perf_stats.xml#perf_stats"/> for usage information and examples.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="show_column_stats">
+
+    <title>SHOW COLUMN STATS Statement</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Performance"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        The <codeph>SHOW TABLE STATS</codeph> and <codeph>SHOW COLUMN STATS</codeph> variants are important for
+        tuning performance and diagnosing performance issues, especially with the largest tables and the most
+        complex join queries.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/security_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/show_security"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following examples show the output of the <codeph>SHOW COLUMN STATS</codeph> statement for some tables,
+        before the <codeph>COMPUTE STATS</codeph> statement is run. Impala deduces some information, such as
+        maximum and average size for fixed-length columns, and leaves and unknown values as <codeph>-1</codeph>.
+      </p>
+
+<codeblock>show column stats customer;
++------------------------+--------+------------------+--------+----------+----------+
+| Column                 | Type   | #Distinct Values | #Nulls | Max Size | Avg Size |
++------------------------+--------+------------------+--------+----------+----------+
+| c_customer_sk          | INT    | -1               | -1     | 4        | 4        |
+| c_customer_id          | STRING | -1               | -1     | -1       | -1       |
+| c_current_cdemo_sk     | INT    | -1               | -1     | 4        | 4        |
+| c_current_hdemo_sk     | INT    | -1               | -1     | 4        | 4        |
+| c_current_addr_sk      | INT    | -1               | -1     | 4        | 4        |
+| c_first_shipto_date_sk | INT    | -1               | -1     | 4        | 4        |
+| c_first_sales_date_sk  | INT    | -1               | -1     | 4        | 4        |
+| c_salutation           | STRING | -1               | -1     | -1       | -1       |
+| c_first_name           | STRING | -1               | -1     | -1       | -1       |
+| c_last_name            | STRING | -1               | -1     | -1       | -1       |
+| c_preferred_cust_flag  | STRING | -1               | -1     | -1       | -1       |
+| c_birth_day            | INT    | -1               | -1     | 4        | 4        |
+| c_birth_month          | INT    | -1               | -1     | 4        | 4        |
+| c_birth_year           | INT    | -1               | -1     | 4        | 4        |
+| c_birth_country        | STRING | -1               | -1     | -1       | -1       |
+| c_login                | STRING | -1               | -1     | -1       | -1       |
+| c_email_address        | STRING | -1               | -1     | -1       | -1       |
+| c_last_review_date     | STRING | -1               | -1     | -1       | -1       |
++------------------------+--------+------------------+--------+----------+----------+
+
+show column stats store_sales;
++-----------------------+-------+------------------+--------+----------+----------+
+| Column                | Type  | #Distinct Values | #Nulls | Max Size | Avg Size |
++-----------------------+-------+------------------+--------+----------+----------+
+| ss_sold_date_sk       | INT   | -1               | -1     | 4        | 4        |
+| ss_sold_time_sk       | INT   | -1               | -1     | 4        | 4        |
+| ss_item_sk            | INT   | -1               | -1     | 4        | 4        |
+| ss_customer_sk        | INT   | -1               | -1     | 4        | 4        |
+| ss_cdemo_sk           | INT   | -1               | -1     | 4        | 4        |
+| ss_hdemo_sk           | INT   | -1               | -1     | 4        | 4        |
+| ss_addr_sk            | INT   | -1               | -1     | 4        | 4        |
+| ss_store_sk           | INT   | -1               | -1     | 4        | 4        |
+| ss_promo_sk           | INT   | -1               | -1     | 4        | 4        |
+| ss_ticket_number      | INT   | -1               | -1     | 4        | 4        |
+| ss_quantity           | INT   | -1               | -1     | 4        | 4        |
+| ss_wholesale_cost     | FLOAT | -1               | -1     | 4        | 4        |
+| ss_list_price         | FLOAT | -1               | -1     | 4        | 4        |
+| ss_sales_price        | FLOAT | -1               | -1     | 4        | 4        |
+| ss_ext_discount_amt   | FLOAT | -1               | -1     | 4        | 4        |
+| ss_ext_sales_price    | FLOAT | -1               | -1     | 4        | 4        |
+| ss_ext_wholesale_cost | FLOAT | -1               | -1     | 4        | 4        |
+| ss_ext_list_price     | FLOAT | -1               | -1     | 4        | 4        |
+| ss_ext_tax            | FLOAT | -1               | -1     | 4        | 4        |
+| ss_coupon_amt         | FLOAT | -1               | -1     | 4        | 4        |
+| ss_net_paid           | FLOAT | -1               | -1     | 4        | 4        |
+| ss_net_paid_inc_tax   | FLOAT | -1               | -1     | 4        | 4        |
+| ss_net_profit         | FLOAT | -1               | -1     | 4        | 4        |
++-----------------------+-------+------------------+--------+----------+----------+
+</codeblock>
+
+      <p>
+        The following examples show the output of the <codeph>SHOW COLUMN STATS</codeph> statement for some tables,
+        after the <codeph>COMPUTE STATS</codeph> statement is run. Now most of the <codeph>-1</codeph> values are
+        changed to reflect the actual table data. The <codeph>#Nulls</codeph> column remains <codeph>-1</codeph>
+        because Impala does not use the number of <codeph>NULL</codeph> values to influence query planning.
+      </p>
+
+<codeblock>compute stats customer;
++------------------------------------------+
+| summary                                  |
++------------------------------------------+
+| Updated 1 partition(s) and 18 column(s). |
++------------------------------------------+
+
+compute stats store_sales;
++------------------------------------------+
+| summary                                  |
++------------------------------------------+
+| Updated 1 partition(s) and 23 column(s). |
++------------------------------------------+
+
+show column stats customer;
++------------------------+--------+------------------+--------+----------+--------+
+| Column                 | Type   | #Distinct Values | #Nulls | Max Size | Avg Size
++------------------------+--------+------------------+--------+----------+--------+
+| c_customer_sk          | INT    | 139017           | -1     | 4        | 4      |
+| c_customer_id          | STRING | 111904           | -1     | 16       | 16     |
+| c_current_cdemo_sk     | INT    | 95837            | -1     | 4        | 4      |
+| c_current_hdemo_sk     | INT    | 8097             | -1     | 4        | 4      |
+| c_current_addr_sk      | INT    | 57334            | -1     | 4        | 4      |
+| c_first_shipto_date_sk | INT    | 4374             | -1     | 4        | 4      |
+| c_first_sales_date_sk  | INT    | 4409             | -1     | 4        | 4      |
+| c_salutation           | STRING | 7                | -1     | 4        | 3.1308 |
+| c_first_name           | STRING | 3887             | -1     | 11       | 5.6356 |
+| c_last_name            | STRING | 4739             | -1     | 13       | 5.9106 |
+| c_preferred_cust_flag  | STRING | 3                | -1     | 1        | 0.9656 |
+| c_birth_day            | INT    | 31               | -1     | 4        | 4      |
+| c_birth_month          | INT    | 12               | -1     | 4        | 4      |
+| c_birth_year           | INT    | 71               | -1     | 4        | 4      |
+| c_birth_country        | STRING | 205              | -1     | 20       | 8.4001 |
+| c_login                | STRING | 1                | -1     | 0        | 0      |
+| c_email_address        | STRING | 94492            | -1     | 46       | 26.485 |
+| c_last_review_date     | STRING | 349              | -1     | 7        | 6.7561 |
++------------------------+--------+------------------+--------+----------+--------+
+
+show column stats store_sales;
++-----------------------+-------+------------------+--------+----------+----------+
+| Column                | Type  | #Distinct Values | #Nulls | Max Size | Avg Size |
++-----------------------+-------+------------------+--------+----------+----------+
+| ss_sold_date_sk       | INT   | 4395             | -1     | 4        | 4        |
+| ss_sold_time_sk       | INT   | 63617            | -1     | 4        | 4        |
+| ss_item_sk            | INT   | 19463            | -1     | 4        | 4        |
+| ss_customer_sk        | INT   | 122720           | -1     | 4        | 4        |
+| ss_cdemo_sk           | INT   | 242982           | -1     | 4        | 4        |
+| ss_hdemo_sk           | INT   | 8097             | -1     | 4        | 4        |
+| ss_addr_sk            | INT   | 70770            | -1     | 4        | 4        |
+| ss_store_sk           | INT   | 6                | -1     | 4        | 4        |
+| ss_promo_sk           | INT   | 355              | -1     | 4        | 4        |
+| ss_ticket_number      | INT   | 304098           | -1     | 4        | 4        |
+| ss_quantity           | INT   | 105              | -1     | 4        | 4        |
+| ss_wholesale_cost     | FLOAT | 9600             | -1     | 4        | 4        |
+| ss_list_price         | FLOAT | 22191            | -1     | 4        | 4        |
+| ss_sales_price        | FLOAT | 20693            | -1     | 4        | 4        |
+| ss_ext_discount_amt   | FLOAT | 228141           | -1     | 4        | 4        |
+| ss_ext_sales_price    | FLOAT | 433550           | -1     | 4        | 4        |
+| ss_ext_wholesale_cost | FLOAT | 406291           | -1     | 4        | 4        |
+| ss_ext_list_price     | FLOAT | 574871           | -1     | 4        | 4        |
+| ss_ext_tax            | FLOAT | 91806            | -1     | 4        | 4        |
+| ss_coupon_amt         | FLOAT | 228141           | -1     | 4        | 4        |
+| ss_net_paid           | FLOAT | 493107           | -1     | 4        | 4        |
+| ss_net_paid_inc_tax   | FLOAT | 653523           | -1     | 4        | 4        |
+| ss_net_profit         | FLOAT | 611934           | -1     | 4        | 4        |
++-----------------------+-------+------------------+--------+----------+----------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+      <p rev="CDH-19187">
+        The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+        typically the <codeph>impala</codeph> user, must have read and execute
+        permissions for all directories that are part of the table.
+        (A table could span multiple different HDFS directories if it is partitioned.
+        The directories could be widely scattered because a partition can reside
+        in an arbitrary HDFS directory based on its <codeph>LOCATION</codeph> attribute.)
+        The Impala user must also have execute
+        permission for the database directory, and any parent directories of the database directory in HDFS.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_compute_stats.xml#compute_stats"/>, <xref href="impala_show.xml#show_table_stats"/>
+      </p>
+
+      <p>
+        See <xref href="impala_perf_stats.xml#perf_stats"/> for usage information and examples.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept rev="1.4.0" id="show_partitions">
+
+    <title>SHOW PARTITIONS Statement</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Schemas"/>
+      <!-- At some point, need to figure out categories related to partitioning. (Partitioned Tables etc.) -->
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        <codeph>SHOW PARTITIONS</codeph> displays information about each partition for a partitioned table. (The
+        output is the same as the <codeph>SHOW TABLE STATS</codeph> statement, but <codeph>SHOW PARTITIONS</codeph>
+        only works on a partitioned table.) Because it displays table statistics for all partitions, the output is
+        more informative if you have run the <codeph>COMPUTE STATS</codeph> statement after creating all the
+        partitions. See <xref href="impala_compute_stats.xml#compute_stats"/> for details. For example, on a
+        <codeph>CENSUS</codeph> table partitioned on the <codeph>YEAR</codeph> column:
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/security_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/show_security"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock rev="1.4.0">[localhost:21000] &gt; show partitions census;
++-------+-------+--------+------+---------+
+| year  | #Rows | #Files | Size | Format  |
++-------+-------+--------+------+---------+
+| 2000  | -1    | 0      | 0B   | TEXT    |
+| 2004  | -1    | 0      | 0B   | TEXT    |
+| 2008  | -1    | 0      | 0B   | TEXT    |
+| 2010  | -1    | 0      | 0B   | TEXT    |
+| 2011  | 4     | 1      | 22B  | TEXT    |
+| 2012  | 4     | 1      | 22B  | TEXT    |
+| 2013  | 1     | 1      | 231B | PARQUET |
+| Total | 9     | 3      | 275B |         |
++-------+-------+--------+------+---------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+      <p rev="CDH-19187">
+        The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+        typically the <codeph>impala</codeph> user, must have read and execute
+        permissions for all directories that are part of the table.
+        (A table could span multiple different HDFS directories if it is partitioned.
+        The directories could be widely scattered because a partition can reside
+        in an arbitrary HDFS directory based on its <codeph>LOCATION</codeph> attribute.)
+        The Impala user must also have execute
+        permission for the database directory, and any parent directories of the database directory in HDFS.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        See <xref href="impala_perf_stats.xml#perf_stats"/> for usage information and examples.
+      </p>
+
+      <p>
+        <xref href="impala_show.xml#show_table_stats"/>, <xref href="impala_partitioning.xml#partitioning"/>
+      </p>
+    </conbody>
+  </concept>
+
+  <concept rev="1.3.0" id="show_functions">
+
+    <title>SHOW FUNCTIONS Statement</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="UDFs"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        By default, <codeph>SHOW FUNCTIONS</codeph> displays user-defined functions (UDFs) and <codeph>SHOW
+        AGGREGATE FUNCTIONS</codeph> displays user-defined aggregate functions (UDAFs) associated with a particular
+        database. The output from <codeph>SHOW FUNCTIONS</codeph> includes the argument signature of each function.
+        You specify this argument signature as part of the <codeph>DROP FUNCTION</codeph> statement. You might have
+        several UDFs with the same name, each accepting different argument data types.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p rev="2.5.0">
+        In <keyword keyref="impala25_full"/> and higher, the <codeph>SHOW FUNCTIONS</codeph> output includes
+        a new column, labelled <codeph>is persistent</codeph>. This property is <codeph>true</codeph> for
+        Impala built-in functions, C++ UDFs, and Java UDFs created using the new <codeph>CREATE FUNCTION</codeph>
+        syntax with no signature. It is <codeph>false</codeph> for Java UDFs created using the old
+        <codeph>CREATE FUNCTION</codeph> syntax that includes the types for the arguments and return value.
+        Any functions with <codeph>false</codeph> shown for this property must be created again by the
+        <codeph>CREATE FUNCTION</codeph> statement each time the Impala catalog server is restarted.
+        See <codeph>CREATE FUNCTION</codeph> for information on switching to the new syntax, so that
+        Java UDFs are preserved across restarts. Java UDFs that are persisted this way are also easier
+        to share across Impala and Hive.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/security_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/show_security"/>
+
+      <p conref="../shared/impala_common.xml#common/permissions_blurb_no"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        To display Impala built-in functions, specify the special database name <codeph>_impala_builtins</codeph>:
+      </p>
+
+<codeblock rev="2.5.0">show functions in _impala_builtins;
++--------------+-------------------------------------------------+-------------+---------------+
+| return type  | signature                                       | binary type | is persistent |
++--------------+-------------------------------------------------+-------------+---------------+
+| BIGINT       | abs(BIGINT)                                     | BUILTIN     | true          |
+| DECIMAL(*,*) | abs(DECIMAL(*,*))                               | BUILTIN     | true          |
+| DOUBLE       | abs(DOUBLE)                                     | BUILTIN     | true          |
+| FLOAT        | abs(FLOAT)                                      | BUILTIN     | true          |
++----------------+----------------------------------------+
+...
+
+show functions in _impala_builtins like '*week*';
++-------------+------------------------------+-------------+---------------+
+| return type | signature                    | binary type | is persistent |
++-------------+------------------------------+-------------+---------------+
+| INT         | dayofweek(TIMESTAMP)         | BUILTIN     | true          |
+| INT         | weekofyear(TIMESTAMP)        | BUILTIN     | true          |
+| TIMESTAMP   | weeks_add(TIMESTAMP, BIGINT) | BUILTIN     | true          |
+| TIMESTAMP   | weeks_add(TIMESTAMP, INT)    | BUILTIN     | true          |
+| TIMESTAMP   | weeks_sub(TIMESTAMP, BIGINT) | BUILTIN     | true          |
+| TIMESTAMP   | weeks_sub(TIMESTAMP, INT)    | BUILTIN     | true          |
++-------------+------------------------------+-------------+---------------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_functions_overview.xml#functions"/>, <xref href="impala_functions.xml#builtins"/>,
+        <xref href="impala_udf.xml#udfs"/>,
+        <xref href="impala_show.xml#show_databases"/>,
+        <xref href="impala_show.xml#show_tables"/>
+      </p>
+    </conbody>
+  </concept>
+
+  <concept rev="someday" audience="Cloudera" id="show_data_sources">
+
+    <title>SHOW DATA SOURCES Statement (CDH x.y and later only)</title>
+
+    <conbody>
+
+      <p>
+        <codeph>SHOW DATA SOURCES</codeph> lists the external data sources defined by the <codeph>CREATE DATA
+        SOURCE</codeph> statement. To show only those names matching a pattern, use the <codeph>LIKE</codeph>
+        clause with asterisks for wildcards, for example <codeph>SHOW DATA SOURCES LIKE '*sql*'</codeph>. These
+        data sources are global, not associated with a specific Impala database, so there is no <codeph>IN</codeph>
+        clause as in most other kinds of objects.
+      </p>
+
+<!--
+<p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>To do: construct example for SHOW DATA SOURCES when that statement is externalized</codeblock>
+-->
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p></p>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_smallint.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_smallint.xml b/docs/topics/impala_smallint.xml
new file mode 100644
index 0000000..1ed95e6
--- /dev/null
+++ b/docs/topics/impala_smallint.xml
@@ -0,0 +1,101 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="smallint">
+
+  <title>SMALLINT Data Type</title>
+  <titlealts audience="PDF"><navtitle>SMALLINT</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      A 2-byte integer data type used in <codeph>CREATE TABLE</codeph> and <codeph>ALTER TABLE</codeph> statements.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p>
+      In the column definition of a <codeph>CREATE TABLE</codeph> statement:
+    </p>
+
+<codeblock><varname>column_name</varname> SMALLINT</codeblock>
+
+    <p>
+      <b>Range:</b> -32768 .. 32767. There is no <codeph>UNSIGNED</codeph> subtype.
+    </p>
+
+    <p>
+      <b>Conversions:</b> Impala automatically converts to a larger integer type (<codeph>INT</codeph> or
+      <codeph>BIGINT</codeph>) or a floating-point type (<codeph>FLOAT</codeph> or <codeph>DOUBLE</codeph>)
+      automatically. Use <codeph>CAST()</codeph> to convert to <codeph>TINYINT</codeph>, <codeph>STRING</codeph>,
+      or <codeph>TIMESTAMP</codeph>.
+      <ph conref="../shared/impala_common.xml#common/cast_int_to_timestamp"/>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      For a convenient and automated way to check the bounds of the <codeph>SMALLINT</codeph> type, call the
+      functions <codeph>MIN_SMALLINT()</codeph> and <codeph>MAX_SMALLINT()</codeph>.
+    </p>
+
+    <p>
+      If an integer value is too large to be represented as a <codeph>SMALLINT</codeph>, use an
+      <codeph>INT</codeph> instead.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/null_bad_numeric_cast"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>CREATE TABLE t1 (x SMALLINT);
+SELECT CAST(1000 AS SMALLINT);
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/parquet_blurb"/>
+
+<!-- Duplicated under TINYINT and SMALLINT. Turn into a conref in both places. -->
+
+    <p rev="1.4.0">
+      Physically, Parquet files represent <codeph>TINYINT</codeph> and <codeph>SMALLINT</codeph> values as 32-bit
+      integers. Although Impala rejects attempts to insert out-of-range values into such columns, if you create a
+      new table with the <codeph>CREATE TABLE ... LIKE PARQUET</codeph> syntax, any <codeph>TINYINT</codeph> or
+      <codeph>SMALLINT</codeph> columns in the original table turn into <codeph>INT</codeph> columns in the new
+      table.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/partitioning_good"/>
+
+    <p conref="../shared/impala_common.xml#common/hbase_ok"/>
+
+    <p conref="../shared/impala_common.xml#common/text_bulky"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/compatibility_blurb"/> -->
+
+    <p conref="../shared/impala_common.xml#common/internals_2_bytes"/>
+
+    <p conref="../shared/impala_common.xml#common/added_forever"/>
+
+    <p conref="../shared/impala_common.xml#common/column_stats_constant"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/restrictions_blurb"/> -->
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_literals.xml#numeric_literals"/>, <xref href="impala_tinyint.xml#tinyint"/>,
+      <xref href="impala_smallint.xml#smallint"/>, <xref href="impala_int.xml#int"/>,
+      <xref href="impala_bigint.xml#bigint"/>, <xref href="impala_decimal.xml#decimal"/>,
+      <xref href="impala_math_functions.xml#math_functions"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_ssl.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_ssl.xml b/docs/topics/impala_ssl.xml
new file mode 100644
index 0000000..1e0f2da
--- /dev/null
+++ b/docs/topics/impala_ssl.xml
@@ -0,0 +1,267 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="ssl">
+
+  <title id="tls">Configuring TLS/SSL for Impala</title>
+
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="SSL"/>
+      <data name="Category" value="Encryption"/>
+      <data name="Category" value="Configuring"/>
+      <data name="Category" value="Administrators"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">SSL</indexterm>
+      Impala supports TLS/SSL network encryption, between Impala and client programs, and between the Impala-related daemons running on
+      different nodes in the cluster. This feature is important when you also use other features such as Kerberos authentication or Sentry
+      authorization, where credentials are being transmitted back and forth.
+      <!-- Formerly: conref="../shared/CDHVariables.xml#xd_583c10bfdbd326ba-3ca24a24-13d80143249- -7f9a/CMCDH_EitherOK" -->
+      <note type="important" id="CMCDH_EitherOK">
+        <ul id="ul_e2s_bcd_np">
+          <li>You can use either Cloudera Manager or the following command-line instructions
+            to complete this configuration.</li>
+          <!-- Took out another too-specific conref, to the CDH minor version also in CDHVariables.xml. -->
+          <li>This information applies specifically to the version of Impala shown in the HTML page header
+            or on the PDF title page.
+            If you use an earlier version of CDH, see the documentation for that version located at
+            <xref href="http://www.cloudera.com/content/support/en/documentation.html" format="html" scope="external">Cloudera Documentation</xref>.</li>
+        </ul></note>
+      />
+    </p>
+
+  </conbody>
+
+  <concept id="concept_gnk_2tt_qp">
+
+    <title>Using Cloudera Manager</title>
+
+    <prolog>
+      <metadata>
+        <data name="Category" value="Cloudera Manager"/>
+      </metadata>
+    </prolog>
+
+    <conbody>
+
+      <p>
+        To configure Impala to listen for Beeswax and HiveServer2 requests on TLS/SSL-secured ports:
+        <ol id="ol_rnf_ftt_qp">
+          <li>
+            Open the Cloudera Manager Admin Console and go to the <uicontrol>Impala</uicontrol> service.
+          </li>
+
+          <!-- Formerly: conref="../shared/cm_common_elements.xml#cm/config_edit" -->
+          <li>Click the <uicontrol>Configuration</uicontrol> tab.</li>
+
+          <li>
+            Select <menucascade><uicontrol>Scope</uicontrol><uicontrol>Impala (Service-Wide)</uicontrol></menucascade>.
+          </li>
+
+          <li>
+            Select <menucascade><uicontrol>Category</uicontrol><uicontrol>Security</uicontrol></menucascade>.
+          </li>
+
+          <li>
+            Edit the following properties:
+            <table frame="all"
+              id="table_drf_ftt_qp">
+              <title>Impala SSL Properties</title>
+              <tgroup cols="2">
+                <colspec colname="c1" colnum="1" colwidth="1*"/>
+                <colspec colname="c2" colnum="2" colwidth="2.5*"/>
+                <thead>
+                  <row>
+                    <entry>
+                      Property
+                    </entry>
+                    <entry>
+                      Description
+                    </entry>
+                  </row>
+                </thead>
+                <tbody>
+                  <row>
+                    <entry>
+                      <b>Enable TLS/SSL for Impala Client Services</b>
+                    </entry>
+                    <entry>
+                      Encrypt communication between clients (like ODBC, JDBC, and the Impala shell) and the Impala daemon using Transport
+                      Layer Security (TLS) (formerly known as Secure Socket Layer (SSL)).
+                    </entry>
+                  </row>
+                  <row>
+                    <entry>
+                      <b>SSL/TLS Certificate for Clients</b>
+                    </entry>
+                    <entry>
+                      Local path to the X509 certificate that identifies the Impala daemon to clients during TLS/SSL connections. This
+                      file must be in PEM format.
+                    </entry>
+                  </row>
+                  <row>
+                    <entry>
+                      <b>SSL/TLS Private Key for Clients</b>
+                    </entry>
+                    <entry>
+                      Local path to the private key that matches the certificate specified in the Certificate for Clients. This file must be
+                      in PEM format.
+                    </entry>
+                  </row>
+                  <row>
+                    <entry>
+                      <b>SSL/TLS Private Key Password for Clients</b>
+                    </entry>
+                    <entry>
+                      A shell command for Impala to run on startup to retrieve the password for a password-protected private key file.
+                      The output of the command is truncated to a maximum of 1024 bytes, and any trailing whitespace (such as spaces
+                      or newline characters) is trimmed. If the command exits with an error, Impala does not start. If the password
+                      is incorrect, clients cannot connect to the server regardless of whether the public key is correct.
+                    </entry>
+                  </row>
+                  <row>
+                    <entry>
+                      <b>SSL/TLS CA Certificate</b>
+                    </entry>
+                    <entry>
+                      Must be specified for TLS/SSL encryption to be enabled for communication
+                      between internal Impala components.
+                    </entry>
+                  </row>
+                  <row>
+                    <entry>
+                      <b>SSL/TLS Certificate for <varname>Impala component</varname> Webserver</b>
+                    </entry>
+                    <entry>
+                      There are three of these configuration settings, one each for <q>Impala Daemon</q>,
+                      <q>Catalog Server</q>, and <q>Statestore</q>.
+                      Each of these Impala components has its own internal web server that powers the
+                      associated web UI with diagnostic information.
+                      The configuration setting represents the local path to the X509 certificate that
+                      identifies the web server to clients during TLS/SSL connections. This
+                      file must be in PEM format.
+                    </entry>
+                  </row>
+                </tbody>
+              </tgroup>
+            </table>
+          </li>
+
+          <!-- Formerly: conref="../shared/cm_common_elements.xml#cm/save_changes_short" -->
+          <li>Click <uicontrol>Save Changes</uicontrol> to commit the changes.</li>
+
+          <li>
+            Restart the Impala service.
+          </li>
+        </ol>
+      </p>
+
+      <p>
+        For information on configuring TLS/SSL communication with the <codeph>impala-shell</codeph> interpreter, see
+        <xref href="#concept_q1p_j2d_rp/secref"/>.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="concept_q1p_j2d_rp">
+
+    <title>Using the Command Line</title>
+
+    <conbody>
+
+<!--
+Info from Henry, from https://docs.google.com/a/cloudera.com/document/d/1u00CJ8WRzXR-1AK_WnQlR6LMtY-7Rc3eHaKNgw3IZvA/edit
+-->
+
+      <p>
+        To enable SSL for when client applications connect to Impala, add both of the following flags to the <cmdname>impalad</cmdname> startup options:
+      </p>
+
+      <ul id="ul_i2p_m2d_rp">
+        <li>
+          <codeph>--ssl_server_certificate</codeph>: the full path to the server certificate, on the local filesystem.
+        </li>
+
+        <li>
+          <codeph>--ssl_private_key</codeph>: the full path to the server private key, on the local filesystem.
+        </li>
+      </ul>
+
+      <p rev="2.3.0">
+        In <keyword keyref="impala23_full"/> and higher, Impala can also use SSL for its own internal communication between the
+        <cmdname>impalad</cmdname>, <codeph>statestored</codeph>, and <codeph>catalogd</codeph> daemons.
+        To enable this additional SSL encryption, set the <codeph>--ssl_server_certificate</codeph>
+        and <codeph>--ssl_private_key</codeph> flags in the startup options for
+        <cmdname>impalad</cmdname>, <cmdname>catalogd</cmdname>, and <cmdname>statestored</cmdname>,
+        and also add the <codeph>--ssl_client_ca_certificate</codeph> flag for all three of those daemons.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/impala_kerberos_ssl_caveat"/>
+
+      <p>
+        If either of these flags are set, both must be set. In that case, Impala starts listening for Beeswax and HiveServer2 requests on
+        SSL-secured ports only. (The port numbers stay the same; see <xref href="impala_ports.xml#ports"/> for details.)
+      </p>
+
+      <p>
+        Since Impala uses passphrase-less certificates in PEM format, you can reuse a host's existing Java keystore by converting it to the
+        PEM format. For instructions, see
+        <xref audience="integrated" href="cm_sg_openssl_jks.xml#concept_ek3_sdl_rp"/><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_openssl_jks.html" scope="external" format="html"/>.
+      </p>
+
+      <section id="secref">
+
+        <title>Configuring TLS/SSL Communication for the Impala Shell</title>
+
+        <p>
+          Typically, a client program has corresponding configuration properties in Cloudera Manager to verify that it is connecting to the
+          right server. For example, with SSL enabled for Impala, you use the following options when starting the
+          <cmdname>impala-shell</cmdname> interpreter:
+        </p>
+
+        <ul id="ul_kgp_m2d_rp">
+          <li>
+            <codeph>--ssl</codeph>: enables TLS/SSL for <cmdname>impala-shell</cmdname>.
+          </li>
+
+          <li>
+            <codeph>--ca_cert</codeph>: the local pathname pointing to the third-party CA certificate, or to a copy of the server
+            certificate for self-signed server certificates.
+          </li>
+        </ul>
+
+        <p>
+          If <codeph>--ca_cert</codeph> is not set, <cmdname>impala-shell</cmdname> enables TLS/SSL, but does not validate the server
+          certificate. This is useful for connecting to a known-good Impala that is only running over TLS/SSL, when a copy of the
+          certificate is not available (such as when debugging customer installations).
+        </p>
+
+      </section>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="ssl_jdbc_odbc">
+    <title>Using TLS/SSL with Business Intelligence Tools</title>
+    <conbody>
+      <p>
+        You can use Kerberos authentication, TLS/SSL encryption, or both to secure
+        connections from JDBC and ODBC applications to Impala.
+        See <xref href="impala_jdbc.xml#impala_jdbc"/> and <xref href="impala_odbc.xml#impala_odbc"/>
+        for details.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/hive_jdbc_ssl_kerberos_caveat"/>
+    </conbody>
+  </concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_stddev.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_stddev.xml b/docs/topics/impala_stddev.xml
new file mode 100644
index 0000000..9cd4126
--- /dev/null
+++ b/docs/topics/impala_stddev.xml
@@ -0,0 +1,118 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.4" id="stddev">
+
+  <title>STDDEV, STDDEV_SAMP, STDDEV_POP Functions</title>
+  <titlealts audience="PDF"><navtitle>STDDEV, STDDEV_SAMP, STDDEV_POP</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="Aggregate Functions"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">stddev() function</indexterm>
+      <indexterm audience="Cloudera">stddev_samp() function</indexterm>
+      <indexterm audience="Cloudera">stddev_pop() function</indexterm>
+      An aggregate function that
+      <xref href="http://en.wikipedia.org/wiki/Standard_deviation" scope="external" format="html">standard
+      deviation</xref> of a set of numbers.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>{ STDDEV | STDDEV_SAMP | STDDEV_POP } ([DISTINCT | ALL] <varname>expression</varname>)</codeblock>
+
+    <p>
+      This function works with any numeric data type.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/former_odd_return_type_string"/>
+
+    <p>
+      This function is typically used in mathematical formulas related to probability distributions.
+    </p>
+
+    <p>
+      The <codeph>STDDEV_POP()</codeph> and <codeph>STDDEV_SAMP()</codeph> functions compute the population
+      standard deviation and sample standard deviation, respectively, of the input values.
+      (<codeph>STDDEV()</codeph> is an alias for <codeph>STDDEV_SAMP()</codeph>.) Both functions evaluate all input
+      rows matched by the query. The difference is that <codeph>STDDEV_SAMP()</codeph> is scaled by
+      <codeph>1/(N-1)</codeph> while <codeph>STDDEV_POP()</codeph> is scaled by <codeph>1/N</codeph>.
+    </p>
+
+    <p>
+      If no input rows match the query, the result of any of these functions is <codeph>NULL</codeph>. If a single
+      input row matches the query, the result of any of these functions is <codeph>"0.0"</codeph>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      This example demonstrates how <codeph>STDDEV()</codeph> and <codeph>STDDEV_SAMP()</codeph> return the same
+      result, while <codeph>STDDEV_POP()</codeph> uses a slightly different calculation to reflect that the input
+      data is considered part of a larger <q>population</q>.
+    </p>
+
+<codeblock>[localhost:21000] &gt; select stddev(score) from test_scores;
++---------------+
+| stddev(score) |
++---------------+
+| 28.5          |
++---------------+
+[localhost:21000] &gt; select stddev_samp(score) from test_scores;
++--------------------+
+| stddev_samp(score) |
++--------------------+
+| 28.5               |
++--------------------+
+[localhost:21000] &gt; select stddev_pop(score) from test_scores;
++-------------------+
+| stddev_pop(score) |
++-------------------+
+| 28.4858           |
++-------------------+
+</codeblock>
+
+    <p>
+      This example demonstrates that, because the return value of these aggregate functions is a
+      <codeph>STRING</codeph>, you must currently convert the result with <codeph>CAST</codeph>.
+    </p>
+
+<codeblock>[localhost:21000] &gt; create table score_stats as select cast(stddev(score) as decimal(7,4)) `standard_deviation`, cast(variance(score) as decimal(7,4)) `variance` from test_scores;
++-------------------+
+| summary           |
++-------------------+
+| Inserted 1 row(s) |
++-------------------+
+[localhost:21000] &gt; desc score_stats;
++--------------------+--------------+---------+
+| name               | type         | comment |
++--------------------+--------------+---------+
+| standard_deviation | decimal(7,4) |         |
+| variance           | decimal(7,4) |         |
++--------------------+--------------+---------+
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/analytic_not_allowed_caveat"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      The <codeph>STDDEV()</codeph>, <codeph>STDDEV_POP()</codeph>, and <codeph>STDDEV_SAMP()</codeph> functions
+      compute the standard deviation (square root of the variance) based on the results of
+      <codeph>VARIANCE()</codeph>, <codeph>VARIANCE_POP()</codeph>, and <codeph>VARIANCE_SAMP()</codeph>
+      respectively. See <xref href="impala_variance.xml#variance"/> for details about the variance property.
+    </p>
+  </conbody>
+</concept>


[22/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_known_issues.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_known_issues.xml b/docs/topics/impala_known_issues.xml
new file mode 100644
index 0000000..e57ec62
--- /dev/null
+++ b/docs/topics/impala_known_issues.xml
@@ -0,0 +1,1812 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="ver" id="known_issues">
+
+  <title><ph audience="standalone">Known Issues and Workarounds in Impala</ph><ph audience="integrated">Apache Impala (incubating) Known Issues</ph></title>
+
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Release Notes"/>
+      <data name="Category" value="Known Issues"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Upgrading"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      The following sections describe known issues and workarounds in Impala, as of the current production release. This page summarizes the
+      most serious or frequently encountered issues in the current release, to help you make planning decisions about installing and
+      upgrading. Any workarounds are listed here. The bug links take you to the Impala issues site, where you can see the diagnosis and
+      whether a fix is in the pipeline.
+    </p>
+
+    <note>
+      The online issue tracking system for Impala contains comprehensive information and is updated in real time. To verify whether an issue
+      you are experiencing has already been reported, or which release an issue is fixed in, search on the
+      <xref href="https://issues.cloudera.org/" scope="external" format="html">issues.cloudera.org JIRA tracker</xref>.
+    </note>
+
+    <p outputclass="toc inpage"/>
+
+    <p>
+      For issues fixed in various Impala releases, see <xref href="impala_fixed_issues.xml#fixed_issues"/>.
+    </p>
+
+<!-- Use as a template for new issues.
+    <concept id="">
+      <title></title>
+      <conbody>
+        <p>
+        </p>
+        <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/" scope="external" format="html"></xref></p>
+        <p><b>Severity:</b> High</p>
+        <p><b>Resolution:</b> </p>
+        <p><b>Workaround:</b> </p>
+      </conbody>
+    </concept>
+
+-->
+
+  </conbody>
+
+<!-- New known issues for CDH 5.5 / Impala 2.3.
+
+Title: Server-to-server SSL and Kerberos do not work together
+Description: If server<->server SSL is enabled (with ssl_client_ca_certificate), and Kerberos auth is used between servers, the cluster will fail to start.
+Upstream & Internal JIRAs: https://issues.cloudera.org/browse/IMPALA-2598
+Severity: Medium.  Server-to-server SSL is practically unusable but this is a new feature.
+Workaround: No known workaround.
+
+Title: Queries may hang on server-to-server exchange errors
+Description: The DataStreamSender::Channel::CloseInternal() does not close the channel on an error. This will cause the node on the other side of the channel to wait indefinitely causing a hang.
+Upstream & Internal JIRAs: https://issues.cloudera.org/browse/IMPALA-2592
+Severity: Low.  This does not occur frequently.
+Workaround: No known workaround.
+
+Title: Catalogd may crash when loading metadata for tables with many partitions, many columns and with incremental stats
+Description: Incremental stats use up about 400 bytes per partition X column.  So for a table with 20K partitions and 100 columns this is about 800 MB.  When serialized this goes past the 2 GB Java array size limit and leads to a catalog crash.
+Upstream & Internal JIRAs: https://issues.cloudera.org/browse/IMPALA-2648, IMPALA-2647, IMPALA-2649.
+Severity: Low.  This does not occur frequently.
+Workaround:  Reduce the number of partitions.
+
+More from: https://issues.cloudera.org/browse/IMPALA-2093?filter=11278&jql=project%20%3D%20IMPALA%20AND%20priority%20in%20(blocker%2C%20critical)%20AND%20status%20in%20(open%2C%20Reopened)%20AND%20labels%20%3D%20correctness%20ORDER%20BY%20priority%20DESC
+
+IMPALA-2093
+Wrong plan of NOT IN aggregate subquery when a constant is used in subquery predicate
+IMPALA-1652
+Incorrect results with basic predicate on CHAR typed column.
+IMPALA-1459
+Incorrect assignment of predicates through an outer join in an inline view.
+IMPALA-2665
+Incorrect assignment of On-clause predicate inside inline view with an outer join.
+IMPALA-2603
+Crash: impala::Coordinator::ValidateCollectionSlots
+IMPALA-2375
+Fix issues with the legacy join and agg nodes using enable_partitioned_hash_join=false and enable_partitioned_aggregation=false
+IMPALA-1862
+Invalid bool value not reported as a scanner error
+IMPALA-1792
+ImpalaODBC: Can not get the value in the SQLGetData(m-x th column) after the SQLBindCol(m th column)
+IMPALA-1578
+Impala incorrectly handles text data when the new line character \n\r is split between different HDFS block
+IMPALA-2643
+Duplicated column in inline view causes dropping null slots during scan
+IMPALA-2005
+A failed CTAS does not drop the table if the insert fails.
+IMPALA-1821
+Casting scenarios with invalid/inconsistent results
+
+Another list from Alex, of correctness problems with predicates; might overlap with ones I already have:
+
+https://issues.cloudera.org/browse/IMPALA-2665 - Already have
+https://issues.cloudera.org/browse/IMPALA-2643 - Already have
+https://issues.cloudera.org/browse/IMPALA-1459 - Already have
+https://issues.cloudera.org/browse/IMPALA-2144 - Don't have
+
+-->
+
+  <concept id="known_issues_crash">
+
+    <title>Impala Known Issues: Crashes and Hangs</title>
+
+    <conbody>
+
+      <p>
+        These issues can cause Impala to quit or become unresponsive.
+      </p>
+
+    </conbody>
+
+    <concept id="IMPALA-3069" rev="IMPALA-3069">
+
+      <title>Setting BATCH_SIZE query option too large can cause a crash</title>
+
+      <conbody>
+
+        <p>
+          Using a value in the millions for the <codeph>BATCH_SIZE</codeph> query option, together with wide rows or large string values in
+          columns, could cause a memory allocation of more than 2 GB resulting in a crash.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-3069" scope="external" format="html">IMPALA-3069</xref>
+        </p>
+
+        <p>
+          <b>Severity:</b> High
+        </p>
+
+        <p><b>Resolution:</b> Fixed in CDH 5.9.0 / Impala 2.7.0.</p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-3441" rev="IMPALA-3441">
+
+      <title></title>
+
+      <conbody>
+
+        <p>
+          Malformed Avro data, such as out-of-bounds integers or values in the wrong format, could cause a crash when queried.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-3441" scope="external" format="html">IMPALA-3441</xref>
+        </p>
+
+        <p>
+          <b>Severity:</b> High
+        </p>
+
+        <p><b>Resolution:</b> Fixed in CDH 5.9.0 / Impala 2.7.0 and CDH 5.8.2 / Impala 2.6.2.</p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-2592" rev="IMPALA-2592">
+
+      <title>Queries may hang on server-to-server exchange errors</title>
+
+      <conbody>
+
+        <p>
+          The <codeph>DataStreamSender::Channel::CloseInternal()</codeph> does not close the channel on an error. This causes the node on
+          the other side of the channel to wait indefinitely, causing a hang.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2592" scope="external" format="html">IMPALA-2592</xref>
+        </p>
+
+        <p>
+          <b>Resolution:</b> Fixed in CDH 5.7.0 / Impala 2.5.0.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-2365" rev="IMPALA-2365">
+
+      <title>Impalad is crashing if udf jar is not available in hdfs location for first time</title>
+
+      <conbody>
+
+        <p>
+          If the JAR file corresponding to a Java UDF is removed from HDFS after the Impala <codeph>CREATE FUNCTION</codeph> statement is
+          issued, the <cmdname>impalad</cmdname> daemon crashes.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2365" scope="external" format="html">IMPALA-2365</xref>
+        </p>
+
+        <p><b>Resolution:</b> Fixed in CDH 5.7.0 / Impala 2.5.0.</p>
+
+      </conbody>
+
+    </concept>
+
+  </concept>
+
+  <concept id="known_issues_performance">
+
+    <title id="ki_performance">Impala Known Issues: Performance</title>
+
+    <conbody>
+
+      <p>
+        These issues involve the performance of operations such as queries or DDL statements.
+      </p>
+
+    </conbody>
+
+    <concept id="IMPALA-1480" rev="IMPALA-1480">
+
+<!-- Not part of Alex's spreadsheet. Spreadsheet has IMPALA-1423 which mentions it's similar to this one but not a duplicate. -->
+
+      <title>Slow DDL statements for tables with large number of partitions</title>
+
+      <conbody>
+
+        <p>
+          DDL statements for tables with a large number of partitions might be slow.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1480" scope="external" format="html"></xref>IMPALA-1480
+        </p>
+
+        <p>
+          <b>Workaround:</b> Run the DDL statement in Hive if the slowness is an issue.
+        </p>
+
+        <p><b>Resolution:</b> Fixed in CDH 5.7.0 / Impala 2.5.0.</p>
+
+      </conbody>
+
+    </concept>
+
+  </concept>
+
+  <concept id="known_issues_usability">
+
+    <title id="ki_usability">Impala Known Issues: Usability</title>
+
+    <conbody>
+
+      <p>
+        These issues affect the convenience of interacting directly with Impala, typically through the Impala shell or Hue.
+      </p>
+
+    </conbody>
+
+    <concept id="IMPALA-3133" rev="IMPALA-3133">
+
+      <title>Unexpected privileges in show output</title>
+
+      <conbody>
+
+        <p>
+          Due to a timing condition in updating cached policy data from Sentry, the <codeph>SHOW</codeph> statements for Sentry roles could
+          sometimes display out-of-date role settings. Because Impala rechecks authorization for each SQL statement, this discrepancy does
+          not represent a security issue for other statements.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-3133" scope="external" format="html">IMPALA-3133</xref>
+        </p>
+
+        <p>
+          <b>Severity:</b> High
+        </p>
+
+        <p>
+          <b>Resolution:</b> Fixes have been issued for some but not all CDH / Impala releases. Check the JIRA for details of fix releases.
+        </p>
+
+        <p><b>Resolution:</b> Fixed in CDH 5.8.0 / Impala 2.6.0 and CDH 5.7.1 / Impala 2.5.1.</p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-1776" rev="IMPALA-1776">
+
+      <title>Less than 100% progress on completed simple SELECT queries</title>
+
+      <conbody>
+
+        <p>
+          Simple <codeph>SELECT</codeph> queries show less than 100% progress even though they are already completed.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1776" scope="external" format="html">IMPALA-1776</xref>
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="concept_lmx_dk5_lx">
+
+      <title>Unexpected column overflow behavior with INT datatypes</title>
+
+      <conbody>
+
+        <p conref="../shared/impala_common.xml#common/int_overflow_behavior" />
+
+        <p>
+          <b>Bug:</b>
+          <xref href="https://issues.cloudera.org/browse/IMPALA-3123"
+            scope="external" format="html">IMPALA-3123</xref>
+        </p>
+
+      </conbody>
+
+    </concept>
+
+  </concept>
+
+  <concept id="known_issues_drivers">
+
+    <title id="ki_drivers">Impala Known Issues: JDBC and ODBC Drivers</title>
+
+    <conbody>
+
+      <p>
+        These issues affect applications that use the JDBC or ODBC APIs, such as business intelligence tools or custom-written applications
+        in languages such as Java or C++.
+      </p>
+
+    </conbody>
+
+    <concept id="IMPALA-1792" rev="IMPALA-1792">
+
+<!-- Not part of Alex's spreadsheet -->
+
+      <title>ImpalaODBC: Can not get the value in the SQLGetData(m-x th column) after the SQLBindCol(m th column)</title>
+
+      <conbody>
+
+        <p>
+          If the ODBC <codeph>SQLGetData</codeph> is called on a series of columns, the function calls must follow the same order as the
+          columns. For example, if data is fetched from column 2 then column 1, the <codeph>SQLGetData</codeph> call for column 1 returns
+          <codeph>NULL</codeph>.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1792" scope="external" format="html">IMPALA-1792</xref>
+        </p>
+
+        <p>
+          <b>Workaround:</b> Fetch columns in the same order they are defined in the table.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+  </concept>
+
+  <concept id="known_issues_security">
+
+    <title id="ki_security">Impala Known Issues: Security</title>
+
+    <conbody>
+
+      <p>
+        These issues relate to security features, such as Kerberos authentication, Sentry authorization, encryption, auditing, and
+        redaction.
+      </p>
+
+    </conbody>
+
+<!-- To do: Hiding for the moment. https://jira.cloudera.com/browse/CDH-38736 reports the issue is fixed. -->
+
+    <concept id="impala-shell_ssl_dependency" audience="Cloudera" rev="impala-shell_ssl_dependency">
+
+      <title>impala-shell requires Python with ssl module</title>
+
+      <conbody>
+
+        <p>
+          On CentOS 5.10 and Oracle Linux 5.11 using the built-in Python 2.4, invoking the <cmdname>impala-shell</cmdname> with the
+          <codeph>--ssl</codeph> option might fail with the following error:
+        </p>
+
+<codeblock>
+Unable to import the python 'ssl' module. It is required for an SSL-secured connection.
+</codeblock>
+
+<!-- No associated IMPALA-* JIRA... It is the internal JIRA CDH-38736. -->
+
+        <p>
+          <b>Severity:</b> Low, workaround available
+        </p>
+
+        <p>
+          <b>Resolution:</b> Customers are less likely to experience this issue over time, because <codeph>ssl</codeph> module is included
+          in newer Python releases packaged with recent Linux releases.
+        </p>
+
+        <p>
+          <b>Workaround:</b> To use SSL with <cmdname>impala-shell</cmdname> on these platform versions, install the <codeph>ssh</codeph>
+          Python module:
+        </p>
+
+<codeblock>
+yum install python-ssl
+</codeblock>
+
+        <p>
+          Then <cmdname>impala-shell</cmdname> can run when using SSL. For example:
+        </p>
+
+<codeblock>
+impala-shell -s impala --ssl --ca_cert /path_to_truststore/truststore.pem
+</codeblock>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="renewable_kerberos_tickets">
+
+<!-- Not part of Alex's spreadsheet. Not associated with a JIRA number AFAIK. -->
+
+      <title>Kerberos tickets must be renewable</title>
+
+      <conbody>
+
+        <p>
+          In a Kerberos environment, the <cmdname>impalad</cmdname> daemon might not start if Kerberos tickets are not renewable.
+        </p>
+
+        <p>
+          <b>Workaround:</b> Configure your KDC to allow tickets to be renewed, and configure <filepath>krb5.conf</filepath> to request
+          renewable tickets.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+<!-- To do: Fixed in 2.5.0, 2.3.2. Commenting out until I see how it can fix into "known issues now fixed" convention.
+     That set of fix releases looks incomplete so probably have to do some detective work with the JIRA.
+     https://issues.cloudera.org/browse/IMPALA-2598
+    <concept id="IMPALA-2598" rev="IMPALA-2598">
+
+      <title>Server-to-server SSL and Kerberos do not work together</title>
+
+      <conbody>
+
+        <p>
+          If SSL is enabled between internal Impala components (with <codeph>ssl_client_ca_certificate</codeph>), and Kerberos
+          authentication is used between servers, the cluster fails to start.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2598" scope="external" format="html">IMPALA-2598</xref>
+        </p>
+
+        <p>
+          <b>Workaround:</b> Do not use the new <codeph>ssl_client_ca_certificate</codeph> setting on Kerberos-enabled clusters until this
+          issue is resolved.
+        </p>
+
+        <p><b>Resolution:</b> Fixed in CDH 5.7.0 / Impala 2.5.0 and CDH 5.5.2 / Impala 2.3.2.</p>
+
+      </conbody>
+
+    </concept>
+-->
+
+  </concept>
+
+<!--
+  <concept id="known_issues_supportability">
+
+    <title id="ki_supportability">Impala Known Issues: Supportability</title>
+
+    <conbody>
+
+      <p>
+        These issues affect the ability to debug and troubleshoot Impala, such as incorrect output in query profiles or the query state
+        shown in monitoring applications.
+      </p>
+
+    </conbody>
+
+  </concept>
+-->
+
+  <concept id="known_issues_resources">
+
+    <title id="ki_resources">Impala Known Issues: Resources</title>
+
+    <conbody>
+
+      <p>
+        These issues involve memory or disk usage, including out-of-memory conditions, the spill-to-disk feature, and resource management
+        features.
+      </p>
+
+    </conbody>
+
+    <concept id="TSB-168">
+
+      <title>Impala catalogd heap issues when upgrading to 5.7</title>
+
+      <conbody>
+
+        <p>
+          The default heap size for Impala <cmdname>catalogd</cmdname> has changed in <keyword keyref="impala25_full"/> and higher:
+        </p>
+
+        <ul>
+          <li>
+            <p>
+              Before 5.7, by default <cmdname>catalogd</cmdname> was using the JVM's default heap size, which is the smaller of 1/4th of the
+              physical memory or 32 GB.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              Starting with CDH 5.7.0, the default <cmdname>catalogd</cmdname> heap size is 4 GB.
+            </p>
+          </li>
+        </ul>
+
+        <p>
+          For example, on a host with 128GB physical memory this will result in catalogd heap decreasing from 32GB to 4GB. This can result
+          in out-of-memory errors in catalogd and leading to query failures.
+        </p>
+
+        <p audience="Cloudera">
+          <b>Bug:</b> <xref href="https://jira.cloudera.com/browse/TSB-168" scope="external" format="html">TSB-168</xref>
+        </p>
+
+        <p>
+          <b>Severity:</b> High
+        </p>
+
+        <p>
+          <b>Workaround:</b> Increase the <cmdname>catalogd</cmdname> memory limit as follows.
+<!-- See <xref href="impala_scalability.xml#scalability_catalog"/> for the procedure. -->
+<!-- Including full details here via conref, for benefit of PDF readers or anyone else
+             who might have trouble seeing or following the link. -->
+        </p>
+
+        <p conref="../shared/impala_common.xml#common/increase_catalogd_heap_size"/>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-3509" rev="IMPALA-3509">
+
+      <title>Breakpad minidumps can be very large when the thread count is high</title>
+
+      <conbody>
+
+        <p>
+          The size of the breakpad minidump files grows linearly with the number of threads. By default, each thread adds 8 KB to the
+          minidump size. Minidump files could consume significant disk space when the daemons have a high number of threads.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-3509" scope="external" format="html">IMPALA-3509</xref>
+        </p>
+
+        <p>
+          <b>Severity:</b> High
+        </p>
+
+        <p>
+          <b>Workaround:</b> Add <codeph>--minidump_size_limit_hint_kb=<varname>size</varname></codeph> to set a soft upper limit on the
+          size of each minidump file. If the minidump file would exceed that limit, Impala reduces the amount of information for each thread
+          from 8 KB to 2 KB. (Full thread information is captured for the first 20 threads, then 2 KB per thread after that.) The minidump
+          file can still grow larger than the <q>hinted</q> size. For example, if you have 10,000 threads, the minidump file can be more
+          than 20 MB.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-3662" rev="IMPALA-3662">
+
+      <title>Parquet scanner memory increase after IMPALA-2736</title>
+
+      <conbody>
+
+        <p>
+          The initial release of <keyword keyref="impala26_full"/> sometimes has a higher peak memory usage than in previous releases while reading
+          Parquet files.
+        </p>
+
+        <p>
+          <keyword keyref="impala26_full"/> addresses the issue IMPALA-2736, which improves the efficiency of Parquet scans by up to 2x. The faster scans
+          may result in a higher peak memory consumption compared to earlier versions of Impala due to the new column-wise row
+          materialization strategy. You are likely to experience higher memory consumption in any of the following scenarios:
+          <ul>
+            <li>
+              <p>
+                Very wide rows due to projecting many columns in a scan.
+              </p>
+            </li>
+
+            <li>
+              <p>
+                Very large rows due to big column values, for example, long strings or nested collections with many items.
+              </p>
+            </li>
+
+            <li>
+              <p>
+                Producer/consumer speed imbalances, leading to more rows being buffered between a scan (producer) and downstream (consumer)
+                plan nodes.
+              </p>
+            </li>
+          </ul>
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-3662" scope="external" format="html">IMPALA-3662</xref>
+        </p>
+
+        <p>
+          <b>Severity:</b> High
+        </p>
+
+        <p>
+          <b>Workaround:</b> The following query options might help to reduce memory consumption in the Parquet scanner:
+          <ul>
+            <li>
+              Reduce the number of scanner threads, for example: <codeph>set num_scanner_threads=30</codeph>
+            </li>
+
+            <li>
+              Reduce the batch size, for example: <codeph>set batch_size=512</codeph>
+            </li>
+
+            <li>
+              Increase the memory limit, for example: <codeph>set mem_limit=64g</codeph>
+            </li>
+          </ul>
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-691" rev="IMPALA-691">
+
+      <title>Process mem limit does not account for the JVM's memory usage</title>
+
+<!-- Supposed to be resolved for Impala 2.3.0. -->
+
+      <conbody>
+
+        <p>
+          Some memory allocated by the JVM used internally by Impala is not counted against the memory limit for the
+          <cmdname>impalad</cmdname> daemon.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-691" scope="external" format="html">IMPALA-691</xref>
+        </p>
+
+        <p>
+          <b>Workaround:</b> To monitor overall memory usage, use the <cmdname>top</cmdname> command, or add the memory figures in the
+          Impala web UI <uicontrol>/memz</uicontrol> tab to JVM memory usage shown on the <uicontrol>/metrics</uicontrol> tab.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-2375" rev="IMPALA-2375">
+
+<!-- Not part of Alex's spreadsheet -->
+
+      <title>Fix issues with the legacy join and agg nodes using --enable_partitioned_hash_join=false and --enable_partitioned_aggregation=false</title>
+
+      <conbody>
+
+        <p></p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2375" scope="external" format="html">IMPALA-2375</xref>
+        </p>
+
+        <p>
+          <b>Workaround:</b> Transition away from the <q>old-style</q> join and aggregation mechanism if practical.
+        </p>
+
+        <p><b>Resolution:</b> Fixed in CDH 5.7.0 / Impala 2.5.0.</p>
+
+      </conbody>
+
+    </concept>
+
+  </concept>
+
+  <concept id="known_issues_correctness">
+
+    <title id="ki_correctness">Impala Known Issues: Correctness</title>
+
+    <conbody>
+
+      <p>
+        These issues can cause incorrect or unexpected results from queries. They typically only arise in very specific circumstances.
+      </p>
+
+    </conbody>
+
+    <concept id="IMPALA-3084" rev="IMPALA-3084">
+
+      <title>Incorrect assignment of NULL checking predicate through an outer join of a nested collection.</title>
+
+      <conbody>
+
+        <p>
+          A query could return wrong results (too many or too few <codeph>NULL</codeph> values) if it referenced an outer-joined nested
+          collection and also contained a null-checking predicate (<codeph>IS NULL</codeph>, <codeph>IS NOT NULL</codeph>, or the
+          <codeph>&lt;=&gt;</codeph> operator) in the <codeph>WHERE</codeph> clause.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-3084" scope="external" format="html">IMPALA-3084</xref>
+        </p>
+
+        <p>
+          <b>Severity:</b> High
+        </p>
+
+        <p><b>Resolution:</b> Fixed in CDH 5.9.0 / Impala 2.7.0.</p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-3094" rev="IMPALA-3094">
+
+      <title>Incorrect result due to constant evaluation in query with outer join</title>
+
+      <conbody>
+
+        <p>
+          An <codeph>OUTER JOIN</codeph> query could omit some expected result rows due to a constant such as <codeph>FALSE</codeph> in
+          another join clause. For example:
+        </p>
+
+<codeblock><![CDATA[
+explain SELECT 1 FROM alltypestiny a1
+  INNER JOIN alltypesagg a2 ON a1.smallint_col = a2.year AND false
+  RIGHT JOIN alltypes a3 ON a1.year = a1.bigint_col;
++---------------------------------------------------------+
+| Explain String                                          |
++---------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=1.00KB VCores=1 |
+|                                                         |
+| 00:EMPTYSET                                             |
++---------------------------------------------------------+
+]]>
+</codeblock>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-3094" scope="external" format="html">IMPALA-3094</xref>
+        </p>
+
+        <p>
+          <b>Severity:</b> High
+        </p>
+
+        <p>
+          <b>Resolution:</b>
+        </p>
+
+        <p>
+          <b>Workaround:</b>
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-3126" rev="IMPALA-3126">
+
+      <title>Incorrect assignment of an inner join On-clause predicate through an outer join.</title>
+
+      <conbody>
+
+        <p>
+          Impala may return incorrect results for queries that have the following properties:
+        </p>
+
+        <ul>
+          <li>
+            <p>
+              There is an INNER JOIN following a series of OUTER JOINs.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              The INNER JOIN has an On-clause with a predicate that references at least two tables that are on the nullable side of the
+              preceding OUTER JOINs.
+            </p>
+          </li>
+        </ul>
+
+        <p>
+          The following query demonstrates the issue:
+        </p>
+
+<codeblock>
+select 1 from functional.alltypes a left outer join
+  functional.alltypes b on a.id = b.id left outer join
+  functional.alltypes c on b.id = c.id right outer join
+  functional.alltypes d on c.id = d.id inner join functional.alltypes e
+on b.int_col = c.int_col;
+</codeblock>
+
+        <p>
+          The following listing shows the incorrect <codeph>EXPLAIN</codeph> plan:
+        </p>
+
+<codeblock><![CDATA[
++-----------------------------------------------------------+
+| Explain String                                            |
++-----------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=480.04MB VCores=4 |
+|                                                           |
+| 14:EXCHANGE [UNPARTITIONED]                               |
+| |                                                         |
+| 08:NESTED LOOP JOIN [CROSS JOIN, BROADCAST]               |
+| |                                                         |
+| |--13:EXCHANGE [BROADCAST]                                |
+| |  |                                                      |
+| |  04:SCAN HDFS [functional.alltypes e]                   |
+| |     partitions=24/24 files=24 size=478.45KB             |
+| |                                                         |
+| 07:HASH JOIN [RIGHT OUTER JOIN, PARTITIONED]              |
+| |  hash predicates: c.id = d.id                           |
+| |  runtime filters: RF000 <- d.id                         |
+| |                                                         |
+| |--12:EXCHANGE [HASH(d.id)]                               |
+| |  |                                                      |
+| |  03:SCAN HDFS [functional.alltypes d]                   |
+| |     partitions=24/24 files=24 size=478.45KB             |
+| |                                                         |
+| 06:HASH JOIN [LEFT OUTER JOIN, PARTITIONED]               |
+| |  hash predicates: b.id = c.id                           |
+| |  other predicates: b.int_col = c.int_col     <--- incorrect placement; should be at node 07 or 08
+| |  runtime filters: RF001 <- c.int_col                    |
+| |                                                         |
+| |--11:EXCHANGE [HASH(c.id)]                               |
+| |  |                                                      |
+| |  02:SCAN HDFS [functional.alltypes c]                   |
+| |     partitions=24/24 files=24 size=478.45KB             |
+| |     runtime filters: RF000 -> c.id                      |
+| |                                                         |
+| 05:HASH JOIN [RIGHT OUTER JOIN, PARTITIONED]              |
+| |  hash predicates: b.id = a.id                           |
+| |  runtime filters: RF002 <- a.id                         |
+| |                                                         |
+| |--10:EXCHANGE [HASH(a.id)]                               |
+| |  |                                                      |
+| |  00:SCAN HDFS [functional.alltypes a]                   |
+| |     partitions=24/24 files=24 size=478.45KB             |
+| |                                                         |
+| 09:EXCHANGE [HASH(b.id)]                                  |
+| |                                                         |
+| 01:SCAN HDFS [functional.alltypes b]                      |
+|    partitions=24/24 files=24 size=478.45KB                |
+|    runtime filters: RF001 -> b.int_col, RF002 -> b.id     |
++-----------------------------------------------------------+
+]]>
+</codeblock>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-3126" scope="external" format="html">IMPALA-3126</xref>
+        </p>
+
+        <p>
+          <b>Severity:</b> High
+        </p>
+
+        <p>
+          <b>Workaround:</b> High
+        </p>
+
+        <p>
+          For some queries, this problem can be worked around by placing the problematic <codeph>ON</codeph> clause predicate in the
+          <codeph>WHERE</codeph> clause instead, or changing the preceding <codeph>OUTER JOIN</codeph>s to <codeph>INNER JOIN</codeph>s (if
+          the <codeph>ON</codeph> clause predicate would discard <codeph>NULL</codeph>s). For example, to fix the problematic query above:
+        </p>
+
+<codeblock><![CDATA[
+select 1 from functional.alltypes a
+  left outer join functional.alltypes b
+    on a.id = b.id
+  left outer join functional.alltypes c
+    on b.id = c.id
+  right outer join functional.alltypes d
+    on c.id = d.id
+  inner join functional.alltypes e
+where b.int_col = c.int_col
+
++-----------------------------------------------------------+
+| Explain String                                            |
++-----------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=480.04MB VCores=4 |
+|                                                           |
+| 14:EXCHANGE [UNPARTITIONED]                               |
+| |                                                         |
+| 08:NESTED LOOP JOIN [CROSS JOIN, BROADCAST]               |
+| |                                                         |
+| |--13:EXCHANGE [BROADCAST]                                |
+| |  |                                                      |
+| |  04:SCAN HDFS [functional.alltypes e]                   |
+| |     partitions=24/24 files=24 size=478.45KB             |
+| |                                                         |
+| 07:HASH JOIN [RIGHT OUTER JOIN, PARTITIONED]              |
+| |  hash predicates: c.id = d.id                           |
+| |  other predicates: b.int_col = c.int_col          <-- correct assignment
+| |  runtime filters: RF000 <- d.id                         |
+| |                                                         |
+| |--12:EXCHANGE [HASH(d.id)]                               |
+| |  |                                                      |
+| |  03:SCAN HDFS [functional.alltypes d]                   |
+| |     partitions=24/24 files=24 size=478.45KB             |
+| |                                                         |
+| 06:HASH JOIN [LEFT OUTER JOIN, PARTITIONED]               |
+| |  hash predicates: b.id = c.id                           |
+| |                                                         |
+| |--11:EXCHANGE [HASH(c.id)]                               |
+| |  |                                                      |
+| |  02:SCAN HDFS [functional.alltypes c]                   |
+| |     partitions=24/24 files=24 size=478.45KB             |
+| |     runtime filters: RF000 -> c.id                      |
+| |                                                         |
+| 05:HASH JOIN [RIGHT OUTER JOIN, PARTITIONED]              |
+| |  hash predicates: b.id = a.id                           |
+| |  runtime filters: RF001 <- a.id                         |
+| |                                                         |
+| |--10:EXCHANGE [HASH(a.id)]                               |
+| |  |                                                      |
+| |  00:SCAN HDFS [functional.alltypes a]                   |
+| |     partitions=24/24 files=24 size=478.45KB             |
+| |                                                         |
+| 09:EXCHANGE [HASH(b.id)]                                  |
+| |                                                         |
+| 01:SCAN HDFS [functional.alltypes b]                      |
+|    partitions=24/24 files=24 size=478.45KB                |
+|    runtime filters: RF001 -> b.id                         |
++-----------------------------------------------------------+
+]]>
+</codeblock>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-3006" rev="IMPALA-3006">
+
+      <title>Impala may use incorrect bit order with BIT_PACKED encoding</title>
+
+      <conbody>
+
+        <p>
+          Parquet <codeph>BIT_PACKED</codeph> encoding as implemented by Impala is LSB first. The parquet standard says it is MSB first.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-3006" scope="external" format="html">IMPALA-3006</xref>
+        </p>
+
+        <p>
+          <b>Severity:</b> High, but rare in practice because BIT_PACKED is infrequently used, is not written by Impala, and is deprecated
+          in Parquet 2.0.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-3082" rev="IMPALA-3082">
+
+      <title>BST between 1972 and 1995</title>
+
+      <conbody>
+
+        <p>
+          The calculation of start and end times for the BST (British Summer Time) time zone could be incorrect between 1972 and 1995.
+          Between 1972 and 1995, BST began and ended at 02:00 GMT on the third Sunday in March (or second Sunday when Easter fell on the
+          third) and fourth Sunday in October. For example, both function calls should return 13, but actually return 12, in a query such
+          as:
+        </p>
+
+<codeblock>
+select
+  extract(from_utc_timestamp(cast('1970-01-01 12:00:00' as timestamp), 'Europe/London'), "hour") summer70start,
+  extract(from_utc_timestamp(cast('1970-12-31 12:00:00' as timestamp), 'Europe/London'), "hour") summer70end;
+</codeblock>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-3082" scope="external" format="html">IMPALA-3082</xref>
+        </p>
+
+        <p>
+          <b>Severity:</b> High
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-1170" rev="IMPALA-1170">
+
+      <title>parse_url() returns incorrect result if @ character in URL</title>
+
+      <conbody>
+
+        <p>
+          If a URL contains an <codeph>@</codeph> character, the <codeph>parse_url()</codeph> function could return an incorrect value for
+          the hostname field.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1170" scope="external" format="html"></xref>IMPALA-1170
+        </p>
+
+        <p><b>Resolution:</b> Fixed in CDH 5.7.0 / Impala 2.5.0 and CDH 5.5.4 / Impala 2.3.4.</p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-2422" rev="IMPALA-2422">
+
+      <title>% escaping does not work correctly when occurs at the end in a LIKE clause</title>
+
+      <conbody>
+
+        <p>
+          If the final character in the RHS argument of a <codeph>LIKE</codeph> operator is an escaped <codeph>\%</codeph> character, it
+          does not match a <codeph>%</codeph> final character of the LHS argument.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2422" scope="external" format="html">IMPALA-2422</xref>
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-397" rev="IMPALA-397">
+
+      <title>ORDER BY rand() does not work.</title>
+
+      <conbody>
+
+        <p>
+          Because the value for <codeph>rand()</codeph> is computed early in a query, using an <codeph>ORDER BY</codeph> expression
+          involving a call to <codeph>rand()</codeph> does not actually randomize the results.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-397" scope="external" format="html">IMPALA-397</xref>
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-2643" rev="IMPALA-2643">
+
+      <title>Duplicated column in inline view causes dropping null slots during scan</title>
+
+      <conbody>
+
+        <p>
+          If the same column is queried twice within a view, <codeph>NULL</codeph> values for that column are omitted. For example, the
+          result of <codeph>COUNT(*)</codeph> on the view could be less than expected.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2643" scope="external" format="html">IMPALA-2643</xref>
+        </p>
+
+        <p>
+          <b>Workaround:</b> Avoid selecting the same column twice within an inline view.
+        </p>
+
+        <p><b>Resolution:</b> Fixed in CDH 5.7.0 / Impala 2.5.0, CDH 5.5.2 / Impala 2.3.2, and CDH 5.4.10 / Impala 2.2.10.</p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-1459" rev="IMPALA-1459">
+
+<!-- Not part of Alex's spreadsheet -->
+
+      <title>Incorrect assignment of predicates through an outer join in an inline view.</title>
+
+      <conbody>
+
+        <p>
+          A query involving an <codeph>OUTER JOIN</codeph> clause where one of the table references is an inline view might apply predicates
+          from the <codeph>ON</codeph> clause incorrectly.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1459" scope="external" format="html">IMPALA-1459</xref>
+        </p>
+
+        <p><b>Resolution:</b> Fixed in CDH 5.7.0 / Impala 2.5.0, CDH 5.5.2 / Impala 2.3.2, and CDH 5.4.9 / Impala 2.2.9.</p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-2603" rev="IMPALA-2603">
+
+      <title>Crash: impala::Coordinator::ValidateCollectionSlots</title>
+
+      <conbody>
+
+        <p>
+          A query could encounter a serious error if includes multiple nested levels of <codeph>INNER JOIN</codeph> clauses involving
+          subqueries.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2603" scope="external" format="html">IMPALA-2603</xref>
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-2665" rev="IMPALA-2665">
+
+      <title>Incorrect assignment of On-clause predicate inside inline view with an outer join.</title>
+
+      <conbody>
+
+        <p>
+          A query might return incorrect results due to wrong predicate assignment in the following scenario:
+        </p>
+
+        <ol>
+          <li>
+            There is an inline view that contains an outer join
+          </li>
+
+          <li>
+            That inline view is joined with another table in the enclosing query block
+          </li>
+
+          <li>
+            That join has an On-clause containing a predicate that only references columns originating from the outer-joined tables inside
+            the inline view
+          </li>
+        </ol>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2665" scope="external" format="html">IMPALA-2665</xref>
+        </p>
+
+        <p><b>Resolution:</b> Fixed in CDH 5.7.0 / Impala 2.5.0, CDH 5.5.2 / Impala 2.3.2, and CDH 5.4.9 / Impala 2.2.9.</p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-2144" rev="IMPALA-2144">
+
+      <title>Wrong assignment of having clause predicate across outer join</title>
+
+      <conbody>
+
+        <p>
+          In an <codeph>OUTER JOIN</codeph> query with a <codeph>HAVING</codeph> clause, the comparison from the <codeph>HAVING</codeph>
+          clause might be applied at the wrong stage of query processing, leading to incorrect results.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2144" scope="external" format="html">IMPALA-2144</xref>
+        </p>
+
+        <p><b>Resolution:</b> Fixed in CDH 5.7.0 / Impala 2.5.0.</p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-2093" rev="IMPALA-2093">
+
+      <title>Wrong plan of NOT IN aggregate subquery when a constant is used in subquery predicate</title>
+
+      <conbody>
+
+        <p>
+          A <codeph>NOT IN</codeph> operator with a subquery that calls an aggregate function, such as <codeph>NOT IN (SELECT
+          SUM(...))</codeph>, could return incorrect results.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2093" scope="external" format="html">IMPALA-2093</xref>
+        </p>
+
+        <p><b>Resolution:</b> Fixed in CDH 5.7.0 / Impala 2.5.0 and CDH 5.5.4 / Impala 2.3.4.</p>
+
+      </conbody>
+
+    </concept>
+
+  </concept>
+
+  <concept id="known_issues_metadata">
+
+    <title id="ki_metadata">Impala Known Issues: Metadata</title>
+
+    <conbody>
+
+      <p>
+        These issues affect how Impala interacts with metadata. They cover areas such as the metastore database, the <codeph>COMPUTE
+        STATS</codeph> statement, and the Impala <cmdname>catalogd</cmdname> daemon.
+      </p>
+
+    </conbody>
+
+    <concept id="IMPALA-2648" rev="IMPALA-2648">
+
+      <title>Catalogd may crash when loading metadata for tables with many partitions, many columns and with incremental stats</title>
+
+      <conbody>
+
+        <p>
+          Incremental stats use up about 400 bytes per partition for each column. For example, for a table with 20K partitions and 100
+          columns, the memory overhead from incremental statistics is about 800 MB. When serialized for transmission across the network,
+          this metadata exceeds the 2 GB Java array size limit and leads to a <codeph>catalogd</codeph> crash.
+        </p>
+
+        <p>
+          <b>Bugs:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2647" scope="external" format="html">IMPALA-2647</xref>,
+          <xref href="https://issues.cloudera.org/browse/IMPALA-2648" scope="external" format="html">IMPALA-2648</xref>,
+          <xref href="https://issues.cloudera.org/browse/IMPALA-2649" scope="external" format="html">IMPALA-2649</xref>
+        </p>
+
+        <p>
+          <b>Workaround:</b> If feasible, compute full stats periodically and avoid computing incremental stats for that table. The
+          scalability of incremental stats computation is a continuing work item.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-1420" rev="IMPALA-1420 2.0.0">
+
+<!-- Not part of Alex's spreadsheet -->
+
+      <title>Can't update stats manually via alter table after upgrading to CDH 5.2</title>
+
+      <conbody>
+
+        <p></p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1420" scope="external" format="html">IMPALA-1420</xref>
+        </p>
+
+        <p>
+          <b>Workaround:</b> On CDH 5.2, when adjusting table statistics manually by setting the <codeph>numRows</codeph>, you must also
+          enable the Boolean property <codeph>STATS_GENERATED_VIA_STATS_TASK</codeph>. For example, use a statement like the following to
+          set both properties with a single <codeph>ALTER TABLE</codeph> statement:
+        </p>
+
+<codeblock>ALTER TABLE <varname>table_name</varname> SET TBLPROPERTIES('numRows'='<varname>new_value</varname>', 'STATS_GENERATED_VIA_STATS_TASK' = 'true');</codeblock>
+
+        <p>
+          <b>Resolution:</b> The underlying cause is the issue
+          <xref href="https://issues.apache.org/jira/browse/HIVE-8648" scope="external" format="html">HIVE-8648</xref> that affects the
+          metastore in Hive 0.13. The workaround is only needed until the fix for this issue is incorporated into a CDH release.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+  </concept>
+
+  <concept id="known_issues_interop">
+
+    <title id="ki_interop">Impala Known Issues: Interoperability</title>
+
+    <conbody>
+
+      <p>
+        These issues affect the ability to interchange data between Impala and other database systems. They cover areas such as data types
+        and file formats.
+      </p>
+
+    </conbody>
+
+<!-- Opened based on CDH-41605. Not part of Alex's spreadsheet AFAIK. -->
+
+    <concept id="CDH-41605">
+
+      <title>DESCRIBE FORMATTED gives error on Avro table</title>
+
+      <conbody>
+
+        <p>
+          This issue can occur either on old Avro tables (created prior to Hive 1.1 / CDH 5.4) or when changing the Avro schema file by
+          adding or removing columns. Columns added to the schema file will not show up in the output of the <codeph>DESCRIBE
+          FORMATTED</codeph> command. Removing columns from the schema file will trigger a <codeph>NullPointerException</codeph>.
+        </p>
+
+        <p>
+          As a workaround, you can use the output of <codeph>SHOW CREATE TABLE</codeph> to drop and recreate the table. This will populate
+          the Hive metastore database with the correct column definitions.
+        </p>
+
+        <note type="warning">
+          Only use this for external tables, or Impala will remove the data files. In case of an internal table, set it to external first:
+<codeblock>
+ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE');
+</codeblock>
+          (The part in parentheses is case sensitive.) Make sure to pick the right choice between internal and external when recreating the
+          table. See <xref href="impala_tables.xml#tables"/> for the differences between internal and external tables.
+        </note>
+
+        <p audience="Cloudera">
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/CDH-41605" scope="external" format="html">CDH-41605</xref>
+        </p>
+
+        <p>
+          <b>Severity:</b> High
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMP-469">
+
+<!-- Not part of Alex's spreadsheet. Perhaps it really is a permanent limitation and nobody is tracking it? -->
+
+      <title>Deviation from Hive behavior: Impala does not do implicit casts between string and numeric and boolean types.</title>
+
+      <conbody>
+
+        <p audience="Cloudera">
+          <b>Cloudera Bug:</b> <xref href="https://jira.cloudera.com/browse/IMP-469" scope="external" format="html"/>; KI added 0.1
+          <i>Cloudera internal only</i>
+        </p>
+
+        <p>
+          <b>Anticipated Resolution</b>: None
+        </p>
+
+        <p>
+          <b>Workaround:</b> Use explicit casts.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMP-175">
+
+<!-- Not part of Alex's spreadsheet. Perhaps it really is a permanent limitation and nobody is tracking it? -->
+
+      <title>Deviation from Hive behavior: Out of range values float/double values are returned as maximum allowed value of type (Hive returns NULL)</title>
+
+      <conbody>
+
+        <p>
+          Impala behavior differs from Hive with respect to out of range float/double values. Out of range values are returned as maximum
+          allowed value of type (Hive returns NULL).
+        </p>
+
+        <p audience="Cloudera">
+          <b>Cloudera Bug:</b> <xref href="https://jira.cloudera.com/browse/IMP-175" scope="external" format="html">IMPALA-175</xref> ; KI
+          added 0.1 <i>Cloudera internal only</i>
+        </p>
+
+        <p>
+          <b>Workaround:</b> None
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="CDH-13199">
+
+<!-- Not part of Alex's spreadsheet. The CDH- prefix makes it an oddball. -->
+
+      <title>Configuration needed for Flume to be compatible with Impala</title>
+
+      <conbody>
+
+        <p>
+          For compatibility with Impala, the value for the Flume HDFS Sink <codeph>hdfs.writeFormat</codeph> must be set to
+          <codeph>Text</codeph>, rather than its default value of <codeph>Writable</codeph>. The <codeph>hdfs.writeFormat</codeph> setting
+          must be changed to <codeph>Text</codeph> before creating data files with Flume; otherwise, those files cannot be read by either
+          Impala or Hive.
+        </p>
+
+        <p>
+          <b>Resolution:</b> This information has been requested to be added to the upstream Flume documentation.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-635" rev="IMPALA-635">
+
+<!-- Not part of Alex's spreadsheet -->
+
+      <title>Avro Scanner fails to parse some schemas</title>
+
+      <conbody>
+
+        <p>
+          Querying certain Avro tables could cause a crash or return no rows, even though Impala could <codeph>DESCRIBE</codeph> the table.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-635" scope="external" format="html">IMPALA-635</xref>
+        </p>
+
+        <p>
+          <b>Workaround:</b> Swap the order of the fields in the schema specification. For example, <codeph>["null", "string"]</codeph>
+          instead of <codeph>["string", "null"]</codeph>.
+        </p>
+
+        <p>
+          <b>Resolution:</b> Not allowing this syntax agrees with the Avro specification, so it may still cause an error even when the
+          crashing issue is resolved.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-1024" rev="IMPALA-1024">
+
+<!-- Not part of Alex's spreadsheet -->
+
+      <title>Impala BE cannot parse Avro schema that contains a trailing semi-colon</title>
+
+      <conbody>
+
+        <p>
+          If an Avro table has a schema definition with a trailing semicolon, Impala encounters an error when the table is queried.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1024" scope="external" format="html">IMPALA-1024</xref>
+        </p>
+
+        <p>
+          <b>Severity:</b> Remove trailing semicolon from the Avro schema.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-2154" rev="IMPALA-2154">
+
+<!-- Not part of Alex's spreadsheet -->
+
+      <title>Fix decompressor to allow parsing gzips with multiple streams</title>
+
+      <conbody>
+
+        <p>
+          Currently, Impala can only read gzipped files containing a single stream. If a gzipped file contains multiple concatenated
+          streams, the Impala query only processes the data from the first stream.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2154" scope="external" format="html">IMPALA-2154</xref>
+        </p>
+
+        <p>
+          <b>Workaround:</b> Use a different gzip tool to compress file to a single stream file.
+        </p>
+
+        <p><b>Resolution:</b> Fixed in CDH 5.7.0 / Impala 2.5.0.</p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-1578" rev="IMPALA-1578">
+
+<!-- Not part of Alex's spreadsheet -->
+
+      <title>Impala incorrectly handles text data when the new line character \n\r is split between different HDFS block</title>
+
+      <conbody>
+
+        <p>
+          If a carriage return / newline pair of characters in a text table is split between HDFS data blocks, Impala incorrectly processes
+          the row following the <codeph>\n\r</codeph> pair twice.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1578" scope="external" format="html">IMPALA-1578</xref>
+        </p>
+
+        <p>
+          <b>Workaround:</b> Use the Parquet format for large volumes of data where practical.
+        </p>
+
+        <p><b>Resolution:</b> Fixed in CDH 5.8.0 / Impala 2.6.0.</p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-1862" rev="IMPALA-1862">
+
+<!-- Not part of Alex's spreadsheet -->
+
+      <title>Invalid bool value not reported as a scanner error</title>
+
+      <conbody>
+
+        <p>
+          In some cases, an invalid <codeph>BOOLEAN</codeph> value read from a table does not produce a warning message about the bad value.
+          The result is still <codeph>NULL</codeph> as expected. Therefore, this is not a query correctness issue, but it could lead to
+          overlooking the presence of invalid data.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1862" scope="external" format="html">IMPALA-1862</xref>
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-1652" rev="IMPALA-1652">
+
+<!-- To do: Isn't this more a correctness issue? -->
+
+      <title>Incorrect results with basic predicate on CHAR typed column.</title>
+
+      <conbody>
+
+        <p>
+          When comparing a <codeph>CHAR</codeph> column value to a string literal, the literal value is not blank-padded and so the
+          comparison might fail when it should match.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1652" scope="external" format="html">IMPALA-1652</xref>
+        </p>
+
+        <p>
+          <b>Workaround:</b> Use the <codeph>RPAD()</codeph> function to blank-pad literals compared with <codeph>CHAR</codeph> columns to
+          the expected length.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+  </concept>
+
+  <concept id="known_issues_limitations">
+
+    <title>Impala Known Issues: Limitations</title>
+
+    <conbody>
+
+      <p>
+        These issues are current limitations of Impala that require evaluation as you plan how to integrate Impala into your data management
+        workflow.
+      </p>
+
+    </conbody>
+
+    <concept id="IMPALA-77" rev="IMPALA-77">
+
+<!-- Not part of Alex's spreadsheet. Perhaps it really is a permanent limitation and nobody is tracking it? -->
+
+      <title>Impala does not support running on clusters with federated namespaces</title>
+
+      <conbody>
+
+        <p>
+          Impala does not support running on clusters with federated namespaces. The <codeph>impalad</codeph> process will not start on a
+          node running such a filesystem based on the <codeph>org.apache.hadoop.fs.viewfs.ViewFs</codeph> class.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-77" scope="external" format="html">IMPALA-77</xref>
+        </p>
+
+        <p>
+          <b>Anticipated Resolution:</b> Limitation
+        </p>
+
+        <p>
+          <b>Workaround:</b> Use standard HDFS on all Impala nodes.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+  </concept>
+
+  <concept id="known_issues_misc">
+
+    <title>Impala Known Issues: Miscellaneous / Older Issues</title>
+
+    <conbody>
+
+      <p>
+        These issues do not fall into one of the above categories or have not been categorized yet.
+      </p>
+
+    </conbody>
+
+    <concept id="IMPALA-2005" rev="IMPALA-2005">
+
+<!-- Not part of Alex's spreadsheet -->
+
+      <title>A failed CTAS does not drop the table if the insert fails.</title>
+
+      <conbody>
+
+        <p>
+          If a <codeph>CREATE TABLE AS SELECT</codeph> operation successfully creates the target table but an error occurs while querying
+          the source table or copying the data, the new table is left behind rather than being dropped.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2005" scope="external" format="html">IMPALA-2005</xref>
+        </p>
+
+        <p>
+          <b>Workaround:</b> Drop the new table manually after a failed <codeph>CREATE TABLE AS SELECT</codeph>.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-1821" rev="IMPALA-1821">
+
+<!-- Not part of Alex's spreadsheet -->
+
+      <title>Casting scenarios with invalid/inconsistent results</title>
+
+      <conbody>
+
+        <p>
+          Using a <codeph>CAST()</codeph> function to convert large literal values to smaller types, or to convert special values such as
+          <codeph>NaN</codeph> or <codeph>Inf</codeph>, produces values not consistent with other database systems. This could lead to
+          unexpected results from queries.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1821" scope="external" format="html">IMPALA-1821</xref>
+        </p>
+
+<!-- <p><b>Workaround:</b> Doublecheck that <codeph>CAST()</codeph> operations work as expect. The issue applies to expressions involving literals, not values read from table columns.</p> -->
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-1619" rev="IMPALA-1619">
+
+<!-- Not part of Alex's spreadsheet -->
+
+      <title>Support individual memory allocations larger than 1 GB</title>
+
+      <conbody>
+
+        <p>
+          The largest single block of memory that Impala can allocate during a query is 1 GiB. Therefore, a query could fail or Impala could
+          crash if a compressed text file resulted in more than 1 GiB of data in uncompressed form, or if a string function such as
+          <codeph>group_concat()</codeph> returned a value greater than 1 GiB.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1619" scope="external" format="html">IMPALA-1619</xref>
+        </p>
+
+        <p><b>Resolution:</b> Fixed in CDH 5.9.0 / Impala 2.7.0 and CDH 5.8.3 / Impala 2.6.3.</p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-941" rev="IMPALA-941">
+
+<!-- Not part of Alex's spreadsheet. Maybe this is interop? -->
+
+      <title>Impala Parser issue when using fully qualified table names that start with a number.</title>
+
+      <conbody>
+
+        <p>
+          A fully qualified table name starting with a number could cause a parsing error. In a name such as <codeph>db.571_market</codeph>,
+          the decimal point followed by digits is interpreted as a floating-point number.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-941" scope="external" format="html">IMPALA-941</xref>
+        </p>
+
+        <p>
+          <b>Workaround:</b> Surround each part of the fully qualified name with backticks (<codeph>``</codeph>).
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMPALA-532" rev="IMPALA-532">
+
+<!-- Not part of Alex's spreadsheet. Perhaps it really is a permanent limitation and nobody is tracking it? -->
+
+      <title>Impala should tolerate bad locale settings</title>
+
+      <conbody>
+
+        <p>
+          If the <codeph>LC_*</codeph> environment variables specify an unsupported locale, Impala does not start.
+        </p>
+
+        <p>
+          <b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-532" scope="external" format="html">IMPALA-532</xref>
+        </p>
+
+        <p>
+          <b>Workaround:</b> Add <codeph>LC_ALL="C"</codeph> to the environment settings for both the Impala daemon and the Statestore
+          daemon. See <xref href="impala_config_options.xml#config_options"/> for details about modifying these environment settings.
+        </p>
+
+        <p>
+          <b>Resolution:</b> Fixing this issue would require an upgrade to Boost 1.47 in the Impala distribution.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="IMP-1203">
+
+<!-- Not part of Alex's spreadsheet. Perhaps it really is a permanent limitation and nobody is tracking it? -->
+
+      <title>Log Level 3 Not Recommended for Impala</title>
+
+      <conbody>
+
+        <p>
+          The extensive logging produced by log level 3 can cause serious performance overhead and capacity issues.
+        </p>
+
+        <p>
+          <b>Workaround:</b> Reduce the log level to its default value of 1, that is, <codeph>GLOG_v=1</codeph>. See
+          <xref href="impala_logging.xml#log_levels"/> for details about the effects of setting different logging levels.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+  </concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_kudu.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_kudu.xml b/docs/topics/impala_kudu.xml
new file mode 100644
index 0000000..c530cc1
--- /dev/null
+++ b/docs/topics/impala_kudu.xml
@@ -0,0 +1,167 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="impala_kudu" rev="kudu">
+
+  <title>Using Impala to Query Kudu Tables</title>
+
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Kudu"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">Kudu</indexterm>
+      You can use Impala to query Kudu tables. This capability allows convenient access to a storage system that is
+      tuned for different kinds of workloads than the default with Impala. The default Impala tables use data files
+      stored on HDFS, which are ideal for bulk loads and queries using full-table scans. In contrast, Kudu can do
+      efficient queries for data organized either in data warehouse style (with full table scans) or for OLTP-style
+      workloads (with key-based lookups for single rows or small ranges of values).
+    </p>
+
+    <p>
+      Certain Impala SQL statements, such as <codeph>UPDATE</codeph> and <codeph>DELETE</codeph>, only work with
+      Kudu tables. These operations were impractical from a performance perspective to perform at large scale on
+      HDFS data, or on HBase tables.
+    </p>
+
+  </conbody>
+
+  <concept id="kudu_benefits">
+
+    <title>Benefits of Using Kudu Tables with Impala</title>
+
+    <conbody>
+
+      <p>
+        The combination of Kudu and Impala works best for tables where scan performance is important, but data
+        arrives continuously, in small batches, or needs to be updated without being completely replaced. In these
+        scenarios (such as for streaming data), it might be impractical to use Parquet tables because Parquet works
+        best with multi-megabyte data files, requiring substantial overhead to replace or reorganize data files to
+        accomodate frequent additions or changes to data. Impala can query Kudu tables with scan performance close
+        to that of Parquet, and Impala can also perform update or delete operations without replacing the entire
+        table contents. You can also use the Kudu API to do ingestion or transformation operations outside of
+        Impala, and Impala can query the current data at any time.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="kudu_primary_key">
+
+    <title>Primary Key Columns for Kudu Tables</title>
+
+    <conbody>
+
+      <p>
+        Kudu tables introduce the notion of primary keys to Impala for the first time. The primary key is made up
+        of one or more columns, whose values are combined and used as a lookup key during queries. These columns
+        cannot contain any <codeph>NULL</codeph> values or any duplicate values, and can never be updated. For a
+        partitioned Kudu table, all the partition key columns must come from the set of primary key columns.
+      </p>
+
+      <p>
+        Impala itself still does not have the notion of unique or non-<codeph>NULL</codeph> constraints. These
+        restrictions on the primary key columns are enforced on the Kudu side.
+      </p>
+
+      <p>
+        The primary key columns must be the first ones specified in the <codeph>CREATE TABLE</codeph> statement.
+        You specify which column or columns make up the primary key in the table properties, rather than through
+        attributes in the column list.
+      </p>
+
+      <p>
+        Kudu can do extra optimizations for queries that refer to the primary key columns in the
+        <codeph>WHERE</codeph> clause. It is not crucial though to include the primary key columns in the
+        <codeph>WHERE</codeph> clause of every query. The benefit is mainly for partitioned tables,
+        which divide the data among various tablet servers based on the distribution of
+        data values in some or all of the primary key columns.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="kudu_dml">
+
+    <title>Impala DML Support for Kudu Tables</title>
+
+    <conbody>
+
+      <p>
+        Impala supports certain DML statements for Kudu tables only. The <codeph>UPDATE</codeph> and
+        <codeph>DELETE</codeph> statements let you modify data within Kudu tables without rewriting substantial
+        amounts of table data.
+      </p>
+
+      <p>
+        The <codeph>INSERT</codeph> statement for Kudu tables honors the unique and non-<codeph>NULL</codeph>
+        requirements for the primary key columns.
+      </p>
+
+      <p>
+        Because Impala and Kudu do not support transactions, the effects of any <codeph>INSERT</codeph>,
+        <codeph>UPDATE</codeph>, or <codeph>DELETE</codeph> statement are immediately visible. For example, you
+        cannot do a sequence of <codeph>UPDATE</codeph> statements and only make the change visible after all the
+        statements are finished. Also, if a DML statement fails partway through, any rows that were already
+        inserted, deleted, or changed remain in the table; there is no rollback mechanism to undo the changes.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="kudu_partitioning">
+
+    <title>Partitioning for Kudu Tables</title>
+
+    <conbody>
+
+      <p>
+        Kudu tables use special mechanisms to evenly distribute data among the underlying tablet servers. Although
+        we refer to such tables as partitioned tables, they are distinguished from traditional Impala partitioned
+        tables by use of different clauses on the <codeph>CREATE TABLE</codeph> statement. Partitioned Kudu tables
+        use <codeph>DISTRIBUTE BY</codeph>, <codeph>HASH</codeph>, <codeph>RANGE</codeph>, and <codeph>SPLIT
+        ROWS</codeph> clauses rather than the traditional <codeph>PARTITIONED BY</codeph> clause. All of the
+        columns involved in these clauses must be primary key columns. These clauses let you specify different ways
+        to divide the data for each column, or even for different value ranges within a column. This flexibility
+        lets you avoid problems with uneven distribution of data, where the partitioning scheme for HDFS tables
+        might result in some partitions being much larger than others. By setting up an effective partitioning
+        scheme for a Kudu table, you can ensure that the work for a query can be parallelized evenly across the
+        hosts in a cluster.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="kudu_performance">
+
+    <title>Impala Query Performance for Kudu Tables</title>
+
+    <conbody>
+
+      <p>
+        For queries involving Kudu tables, Impala can delegate much of the work of filtering the result set to
+        Kudu, avoiding some of the I/O involved in full table scans of tables containing HDFS data files. This type
+        of optimization is especially effective for partitioned Kudu tables, where the Impala query
+        <codeph>WHERE</codeph> clause refers to one or more primary key columns that are also used as partition key
+        columns. For example, if a partitioned Kudu table uses a <codeph>HASH</codeph> clause for
+        <codeph>col1</codeph> and a <codeph>RANGE</codeph> clause for <codeph>col2</codeph>, a query using a clause
+        such as <codeph>WHERE col1 IN (1,2,3) AND col2 &gt; 100</codeph> can determine exactly which tablet servers
+        contain relevant data, and therefore parallelize the query very efficiently.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_langref.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_langref.xml b/docs/topics/impala_langref.xml
new file mode 100644
index 0000000..f81b76f
--- /dev/null
+++ b/docs/topics/impala_langref.xml
@@ -0,0 +1,74 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="langref">
+
+  <title>Impala SQL Language Reference</title>
+  <titlealts audience="PDF"><navtitle>SQL Reference</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="impala-shell"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Impala uses SQL as its query language. To protect user investment in skills development and query
+      design, Impala provides a high degree of compatibility with the Hive Query Language (HiveQL):
+    </p>
+
+    <ul>
+      <li>
+        Because Impala uses the same metadata store as Hive to record information about table structure and
+        properties, Impala can access tables defined through the native Impala <codeph>CREATE TABLE</codeph>
+        command, or tables created using the Hive data definition language (DDL).
+      </li>
+
+      <li>
+        Impala supports data manipulation (DML) statements similar to the DML component of HiveQL.
+      </li>
+
+      <li>
+        Impala provides many <xref href="impala_functions.xml#builtins">built-in functions</xref> with the same
+        names and parameter types as their HiveQL equivalents.
+      </li>
+    </ul>
+
+    <p>
+      Impala supports most of the same <xref href="impala_langref_sql.xml#langref_sql">statements and
+      clauses</xref> as HiveQL, including, but not limited to <codeph>JOIN</codeph>, <codeph>AGGREGATE</codeph>,
+      <codeph>DISTINCT</codeph>, <codeph>UNION ALL</codeph>, <codeph>ORDER BY</codeph>, <codeph>LIMIT</codeph> and
+      (uncorrelated) subquery in the <codeph>FROM</codeph> clause. Impala also supports <codeph>INSERT
+      INTO</codeph> and <codeph>INSERT OVERWRITE</codeph>.
+    </p>
+
+    <p>
+      Impala supports data types with the same names and semantics as the equivalent Hive data types:
+      <codeph>STRING</codeph>, <codeph>TINYINT</codeph>, <codeph>SMALLINT</codeph>, <codeph>INT</codeph>,
+      <codeph>BIGINT</codeph>, <codeph>FLOAT</codeph>, <codeph>DOUBLE</codeph>, <codeph>BOOLEAN</codeph>,
+      <codeph>STRING</codeph>, <codeph>TIMESTAMP</codeph>.
+    </p>
+
+    <p>
+      For full details about Impala SQL syntax and semantics, see
+      <xref href="impala_langref_sql.xml#langref_sql"/>.
+    </p>
+
+    <p>
+      Most HiveQL <codeph>SELECT</codeph> and <codeph>INSERT</codeph> statements run unmodified with Impala. For
+      information about Hive syntax not available in Impala, see
+      <xref href="impala_langref_unsupported.xml#langref_hiveql_delta"/>.
+    </p>
+
+    <p>
+      For a list of the built-in functions available in Impala queries, see
+      <xref href="impala_functions.xml#builtins"/>.
+    </p>
+
+    <p outputclass="toc"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_langref_sql.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_langref_sql.xml b/docs/topics/impala_langref_sql.xml
new file mode 100644
index 0000000..18b6726
--- /dev/null
+++ b/docs/topics/impala_langref_sql.xml
@@ -0,0 +1,35 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="langref_sql">
+
+  <title>Impala SQL Statements</title>
+  <titlealts audience="PDF"><navtitle>SQL Statements</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      The Impala SQL dialect supports a range of standard elements, plus some extensions for Big Data use cases
+      related to data loading and data warehousing.
+    </p>
+
+    <note>
+      <p>
+        In the <cmdname>impala-shell</cmdname> interpreter, a semicolon at the end of each statement is required.
+        Since the semicolon is not actually part of the SQL syntax, we do not include it in the syntax definition
+        of each statement, but we do show it in examples intended to be run in <cmdname>impala-shell</cmdname>.
+      </p>
+    </note>
+
+    <p audience="PDF" outputclass="toc all">
+      The following sections show the major SQL statements that you work with in Impala:
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_langref_unsupported.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_langref_unsupported.xml b/docs/topics/impala_langref_unsupported.xml
new file mode 100644
index 0000000..82910d6
--- /dev/null
+++ b/docs/topics/impala_langref_unsupported.xml
@@ -0,0 +1,312 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="langref_hiveql_delta">
+
+  <title>SQL Differences Between Impala and Hive</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Hive"/>
+      <data name="Category" value="Porting"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">Hive</indexterm>
+      <indexterm audience="Cloudera">HiveQL</indexterm>
+      Impala's SQL syntax follows the SQL-92 standard, and includes many industry extensions in areas such as
+      built-in functions. See <xref href="impala_porting.xml#porting"/> for a general discussion of adapting SQL
+      code from a variety of database systems to Impala.
+    </p>
+
+    <p>
+      Because Impala and Hive share the same metastore database and their tables are often used interchangeably,
+      the following section covers differences between Impala and Hive in detail.
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="langref_hiveql_unsupported">
+
+    <title>HiveQL Features not Available in Impala</title>
+
+    <conbody>
+
+      <p>
+        The current release of Impala does not support the following SQL features that you might be familiar with
+        from HiveQL:
+      </p>
+
+      <!-- To do:
+        Yeesh, too many separate lists of unsupported Hive syntax.
+        Here, the FAQ, and in some of the intro topics.
+        Some discussion in IMP-1061 about how best to reorg.
+        Lots of opportunities for conrefs.
+      -->
+
+      <ul>
+<!-- Now supported in <keyword keyref="impala23_full"/> and higher. Find places on this page (like already done under lateral views) to note the new data type support.
+        <li>
+          Non-scalar data types such as maps, arrays, structs.
+        </li>
+-->
+
+        <li rev="1.2">
+          Extensibility mechanisms such as <codeph>TRANSFORM</codeph>, custom file formats, or custom SerDes.
+        </li>
+
+        <li rev="CDH-41376">
+          The <codeph>DATE</codeph> data type.
+        </li>
+
+        <li>
+          XML and JSON functions.
+        </li>
+
+        <li>
+          Certain aggregate functions from HiveQL: <codeph>covar_pop</codeph>, <codeph>covar_samp</codeph>,
+          <codeph>corr</codeph>, <codeph>percentile</codeph>, <codeph>percentile_approx</codeph>,
+          <codeph>histogram_numeric</codeph>, <codeph>collect_set</codeph>; Impala supports the set of aggregate
+          functions listed in <xref href="impala_aggregate_functions.xml#aggregate_functions"/> and analytic
+          functions listed in <xref href="impala_analytic_functions.xml#analytic_functions"/>.
+        </li>
+
+        <li>
+          Sampling.
+        </li>
+
+        <li>
+          Lateral views. In <keyword keyref="impala23_full"/> and higher, Impala supports queries on complex types
+          (<codeph>STRUCT</codeph>, <codeph>ARRAY</codeph>, or <codeph>MAP</codeph>), using join notation
+          rather than the <codeph>EXPLODE()</codeph> keyword.
+          See <xref href="impala_complex_types.xml#complex_types"/> for details about Impala support for complex types.
+        </li>
+
+        <li>
+          Multiple <codeph>DISTINCT</codeph> clauses per query, although Impala includes some workarounds for this
+          limitation.
+          <note conref="../shared/impala_common.xml#common/multiple_count_distinct"/>
+        </li>
+      </ul>
+
+      <p>
+        User-defined functions (UDFs) are supported starting in Impala 1.2. See <xref href="impala_udf.xml#udfs"/>
+        for full details on Impala UDFs.
+        <ul>
+          <li>
+            <p>
+              Impala supports high-performance UDFs written in C++, as well as reusing some Java-based Hive UDFs.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              Impala supports scalar UDFs and user-defined aggregate functions (UDAFs). Impala does not currently
+              support user-defined table generating functions (UDTFs).
+            </p>
+          </li>
+
+          <li>
+            <p>
+              Only Impala-supported column types are supported in Java-based UDFs.
+            </p>
+          </li>
+
+          <li>
+            <p conref="../shared/impala_common.xml#common/current_user_caveat"/>
+          </li>
+        </ul>
+      </p>
+
+      <p>
+        Impala does not currently support these HiveQL statements:
+      </p>
+
+      <ul>
+        <li>
+          <codeph>ANALYZE TABLE</codeph> (the Impala equivalent is <codeph>COMPUTE STATS</codeph>)
+        </li>
+
+        <li>
+          <codeph>DESCRIBE COLUMN</codeph>
+        </li>
+
+        <li>
+          <codeph>DESCRIBE DATABASE</codeph>
+        </li>
+
+        <li>
+          <codeph>EXPORT TABLE</codeph>
+        </li>
+
+        <li>
+          <codeph>IMPORT TABLE</codeph>
+        </li>
+
+        <li>
+          <codeph>SHOW TABLE EXTENDED</codeph>
+        </li>
+
+        <li>
+          <codeph>SHOW INDEXES</codeph>
+        </li>
+
+        <li>
+          <codeph>SHOW COLUMNS</codeph>
+        </li>
+
+        <li rev="DOCS-656">
+          <codeph>INSERT OVERWRITE DIRECTORY</codeph>; use <codeph>INSERT OVERWRITE <varname>table_name</varname></codeph>
+          or <codeph>CREATE TABLE AS SELECT</codeph> to materialize query results into the HDFS directory associated
+          with an Impala table.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="langref_hiveql_semantics">
+
+    <title>Semantic Differences Between Impala and HiveQL Features</title>
+
+    <conbody>
+
+      <p>
+        This section covers instances where Impala and Hive have similar functionality, sometimes including the
+        same syntax, but there are differences in the runtime semantics of those features.
+      </p>
+
+      <p>
+        <b>Security:</b>
+      </p>
+
+      <p>
+        Impala utilizes the <xref href="http://sentry.incubator.apache.org/" scope="external" format="html">Apache
+        Sentry </xref> authorization framework, which provides fine-grained role-based access control
+        to protect data against unauthorized access or tampering.
+      </p>
+
+      <p>
+        The Hive component included in <ph rev="upstream">CDH 5.1</ph> and higher now includes Sentry-enabled <codeph>GRANT</codeph>,
+        <codeph>REVOKE</codeph>, and <codeph>CREATE/DROP ROLE</codeph> statements. Earlier Hive releases had a
+        privilege system with <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements that were primarily
+        intended to prevent accidental deletion of data, rather than a security mechanism to protect against
+        malicious users.
+      </p>
+
+      <p>
+        Impala can make use of privileges set up through Hive <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements.
+        Impala has its own <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements in Impala 2.0 and higher.
+        See <xref href="impala_authorization.xml#authorization"/> for the details of authorization in Impala, including
+        how to switch from the original policy file-based privilege model to the Sentry service using privileges
+        stored in the metastore database.
+      </p>
+
+      <p>
+        <b>SQL statements and clauses:</b>
+      </p>
+
+      <p>
+        The semantics of Impala SQL statements varies from HiveQL in some cases where they use similar SQL
+        statement and clause names:
+      </p>
+
+      <ul>
+        <li>
+          Impala uses different syntax and names for query hints, <codeph>[SHUFFLE]</codeph> and
+          <codeph>[NOSHUFFLE]</codeph> rather than <codeph>MapJoin</codeph> or <codeph>StreamJoin</codeph>. See
+          <xref href="impala_joins.xml#joins"/> for the Impala details.
+        </li>
+
+        <li>
+          Impala does not expose MapReduce specific features of <codeph>SORT BY</codeph>, <codeph>DISTRIBUTE
+          BY</codeph>, or <codeph>CLUSTER BY</codeph>.
+        </li>
+
+        <li>
+          Impala does not require queries to include a <codeph>FROM</codeph> clause.
+        </li>
+      </ul>
+
+      <p>
+        <b>Data types:</b>
+      </p>
+
+      <ul>
+        <li>
+          Impala supports a limited set of implicit casts. This can help avoid undesired results from unexpected
+          casting behavior.
+          <ul>
+            <li>
+              Impala does not implicitly cast between string and numeric or Boolean types. Always use
+              <codeph>CAST()</codeph> for these conversions.
+            </li>
+
+            <li>
+              Impala does perform implicit casts among the numeric types, when going from a smaller or less precise
+              type to a larger or more precise one. For example, Impala will implicitly convert a
+              <codeph>SMALLINT</codeph> to a <codeph>BIGINT</codeph> or <codeph>FLOAT</codeph>, but to convert from
+              <codeph>DOUBLE</codeph> to <codeph>FLOAT</codeph> or <codeph>INT</codeph> to <codeph>TINYINT</codeph>
+              requires a call to <codeph>CAST()</codeph> in the query.
+            </li>
+
+            <li>
+              Impala does perform implicit casts from string to timestamp. Impala has a restricted set of literal
+              formats for the <codeph>TIMESTAMP</codeph> data type and the <codeph>from_unixtime()</codeph> format
+              string; see <xref href="impala_timestamp.xml#timestamp"/> for details.
+            </li>
+          </ul>
+          <p>
+            See <xref href="impala_datatypes.xml#datatypes"/> for full details on implicit and explicit casting for
+            all types, and <xref href="impala_conversion_functions.xml#conversion_functions"/> for details about
+            the <codeph>CAST()</codeph> function.
+          </p>
+        </li>
+
+        <li>
+          Impala does not store or interpret timestamps using the local timezone, to avoid undesired results from
+          unexpected time zone issues. Timestamps are stored and interpreted relative to UTC. This difference can
+          produce different results for some calls to similarly named date/time functions between Impala and Hive.
+          See <xref href="impala_datetime_functions.xml#datetime_functions"/> for details about the Impala
+          functions. See <xref href="impala_timestamp.xml#timestamp"/> for a discussion of how Impala handles
+          time zones, and configuration options you can use to make Impala match the Hive behavior more closely
+          when dealing with Parquet-encoded <codeph>TIMESTAMP</codeph> data or when converting between
+          the local time zone and UTC.
+        </li>
+
+        <li>
+          The Impala <codeph>TIMESTAMP</codeph> type can represent dates ranging from 1400-01-01 to 9999-12-31.
+          This is different from the Hive date range, which is 0000-01-01 to 9999-12-31.
+        </li>
+
+        <li>
+          <p conref="../shared/impala_common.xml#common/int_overflow_behavior"/>
+        </li>
+
+      </ul>
+
+      <p>
+        <b>Miscellaneous features:</b>
+      </p>
+
+      <ul>
+        <li>
+          Impala does not provide virtual columns.
+        </li>
+
+        <li>
+          Impala does not expose locking.
+        </li>
+
+        <li>
+          Impala does not expose some configuration properties.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+</concept>



[49/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/impala_html.ditaval
----------------------------------------------------------------------
diff --git a/docs/impala_html.ditaval b/docs/impala_html.ditaval
new file mode 100644
index 0000000..b9403f6
--- /dev/null
+++ b/docs/impala_html.ditaval
@@ -0,0 +1,21 @@
+<?xml version="1.0" encoding="UTF-8"?><val>
+<!-- Exclude Cloudera-only content. This is typically material that's permanently hidden,
+     e.g. obsolete or abandoned. Use pre-release for material being actively worked on
+     that's not ready for prime time. -->
+<prop att="audience" val="Cloudera" action="exclude"/>
+<!-- These two are backward: things marked HTML are excluded from the HTML and
+     things marked PDF are excluded from the PDF. -->
+<prop att="audience" val="HTML" action="exclude"/>
+<prop att="audience" val="PDF" action="include"/>
+<!-- standalone = upstream Impala docs, not part of any larger library
+     integrated = any xrefs, topicrefs, or other residue from original downstream docs
+                  that don't resolve properly in the upstream context -->
+<prop att="audience" val="integrated" action="exclude"/>
+<prop att="audience" val="standalone" action="include"/>
+<!-- John added this so he can work on Impala_Next in master without fear that
+     it will show up too early in released docs -->
+<prop att="audience" val="impala_next" action="exclude"/>
+<!-- This DITAVAL specifically EXCLUDES things marked pre-release -->
+<!-- It is safe to use for generating public artifacts. -->
+<prop att="audience" val="pre-release" action="exclude"/>
+</val>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/impala_pdf.ditaval
----------------------------------------------------------------------
diff --git a/docs/impala_pdf.ditaval b/docs/impala_pdf.ditaval
new file mode 100644
index 0000000..c0f3b8e
--- /dev/null
+++ b/docs/impala_pdf.ditaval
@@ -0,0 +1,21 @@
+<?xml version="1.0" encoding="UTF-8"?><val>
+<!-- Exclude Cloudera-only content. This is typically material that's permanently hidden,
+     e.g. obsolete or abandoned. Use pre-release for material being actively worked on
+     that's not ready for prime time. -->
+  <prop att="audience" val="Cloudera" action="exclude"/>
+<!-- These two are backward: things marked HTML are excluded from the HTML and
+     things marked PDF are excluded from the PDF. -->
+	<prop att="audience" val="PDF" action="exclude"/>
+	<prop att="audience" val="HTML" action="include"/>
+<!-- standalone = upstream Impala docs, not part of any larger library
+     integrated = any xrefs, topicrefs, or other residue from original downstream docs
+                  that don't resolve properly in the upstream context -->
+	<prop att="audience" val="integrated" action="exclude"/>
+	<prop att="audience" val="standalone" action="include"/>
+<!-- John added this so he can work on Impala_Next in master without fear that
+     it will show up too early in released docs -->
+  <prop att="audience" val="impala_next" action="exclude"/>
+<!-- This DITAVAL specifically EXCLUDES things marked pre-release -->
+<!-- It is safe to use for generating public artifacts. -->
+  <prop att="audience" val="pre-release" action="exclude"/>
+</val>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/impala_sqlref.ditamap
----------------------------------------------------------------------
diff --git a/docs/impala_sqlref.ditamap b/docs/impala_sqlref.ditamap
new file mode 100644
index 0000000..608ce49
--- /dev/null
+++ b/docs/impala_sqlref.ditamap
@@ -0,0 +1,146 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE map PUBLIC "-//OASIS//DTD DITA Map//EN" "map.dtd">
+<map id="impala_sqlref">
+  <title>Impala SQL Reference</title>
+  <topicmeta>
+    <prodinfo conref="shared/ImpalaVariables.xml#impala_vars/prodinfo_for_html">
+      <prodname/>
+      <vrmlist>
+        <vrm version="version_dlq_gry_sm"/>
+      </vrmlist>
+    </prodinfo>
+  </topicmeta>
+    <topicref href="topics/impala_langref.xml"/>
+    <topicref href="topics/impala_comments.xml"/>
+    <topicref href="topics/impala_datatypes.xml">
+      <topicref href="topics/impala_array.xml"/>
+      <topicref href="topics/impala_bigint.xml"/>
+      <topicref href="topics/impala_boolean.xml"/>
+      <topicref href="topics/impala_char.xml"/>
+      <topicref href="topics/impala_decimal.xml"/>
+      <topicref href="topics/impala_double.xml"/>
+      <topicref href="topics/impala_float.xml"/>
+      <topicref href="topics/impala_int.xml"/>
+      <topicref href="topics/impala_map.xml"/>
+      <topicref href="topics/impala_real.xml"/>
+      <topicref href="topics/impala_smallint.xml"/>
+      <topicref href="topics/impala_string.xml"/>
+      <topicref href="topics/impala_struct.xml"/>
+      <topicref href="topics/impala_timestamp.xml"/>
+      <topicref href="topics/impala_tinyint.xml"/>
+      <topicref href="topics/impala_varchar.xml"/>
+      <topicref href="topics/impala_complex_types.xml"/>
+    </topicref>
+    <topicref href="topics/impala_literals.xml"/>
+    <topicref href="topics/impala_operators.xml"/>
+    <topicref href="topics/impala_schema_objects.xml">
+      <topicref href="topics/impala_aliases.xml"/>
+      <topicref href="topics/impala_databases.xml"/>
+      <topicref href="topics/impala_functions_overview.xml"/>
+      <topicref href="topics/impala_identifiers.xml"/>
+      <topicref href="topics/impala_tables.xml"/>
+      <topicref href="topics/impala_views.xml"/>
+    </topicref>
+    <topicref href="topics/impala_langref_sql.xml">
+      <topicref href="topics/impala_ddl.xml"/>
+      <topicref href="topics/impala_dml.xml"/>
+      <topicref href="topics/impala_alter_table.xml"/>
+      <topicref href="topics/impala_alter_view.xml"/>
+      <topicref href="topics/impala_compute_stats.xml"/>
+      <topicref href="topics/impala_create_database.xml"/>
+      <topicref href="topics/impala_create_function.xml"/>
+      <topicref href="topics/impala_create_role.xml"/>
+      <topicref href="topics/impala_create_table.xml"/>
+      <topicref href="topics/impala_create_view.xml"/>
+      <topicref audience="impala_next" href="topics/impala_delete.xml"/>
+      <topicref href="topics/impala_describe.xml"/>
+      <topicref href="topics/impala_drop_database.xml"/>
+      <topicref href="topics/impala_drop_function.xml"/>
+      <topicref href="topics/impala_drop_role.xml"/>
+      <topicref href="topics/impala_drop_stats.xml"/>
+      <topicref href="topics/impala_drop_table.xml"/>
+      <topicref href="topics/impala_drop_view.xml"/>
+      <topicref href="topics/impala_explain.xml"/>
+      <topicref href="topics/impala_grant.xml"/>
+      <topicref href="topics/impala_insert.xml"/>
+      <topicref href="topics/impala_invalidate_metadata.xml"/>
+      <topicref href="topics/impala_load_data.xml"/>
+      <topicref href="topics/impala_refresh.xml"/>
+      <topicref href="topics/impala_revoke.xml"/>
+      <topicref href="topics/impala_select.xml">
+        <topicref href="topics/impala_joins.xml"/>
+        <topicref href="topics/impala_order_by.xml"/>
+        <topicref href="topics/impala_group_by.xml"/>
+        <topicref href="topics/impala_having.xml"/>
+        <topicref href="topics/impala_limit.xml"/>
+        <topicref href="topics/impala_offset.xml"/>
+        <topicref href="topics/impala_union.xml"/>
+        <topicref href="topics/impala_subqueries.xml"/>
+        <topicref href="topics/impala_with.xml"/>
+        <topicref href="topics/impala_distinct.xml"/>
+        <topicref href="topics/impala_hints.xml"/>
+      </topicref>
+      <topicref href="topics/impala_set.xml"/>
+        <topicref href="topics/impala_query_options.xml">
+          <topicref href="topics/impala_abort_on_default_limit_exceeded.xml"/>
+          <topicref href="topics/impala_abort_on_error.xml"/>
+          <topicref href="topics/impala_allow_unsupported_formats.xml"/>
+          <topicref href="topics/impala_appx_count_distinct.xml"/>
+          <topicref href="topics/impala_batch_size.xml"/>
+          <topicref href="topics/impala_compression_codec.xml"/>
+          <topicref href="topics/impala_debug_action.xml"/>
+          <topicref href="topics/impala_default_order_by_limit.xml"/>
+          <topicref href="topics/impala_disable_codegen.xml"/>
+          <topicref href="topics/impala_disable_unsafe_spills.xml"/>
+          <topicref href="topics/impala_exec_single_node_rows_threshold.xml"/>
+          <topicref href="topics/impala_explain_level.xml"/>
+          <topicref href="topics/impala_hbase_cache_blocks.xml"/>
+          <topicref href="topics/impala_hbase_caching.xml"/>
+          <topicref href="topics/impala_live_progress.xml"/>
+          <topicref href="topics/impala_live_summary.xml"/>
+          <topicref href="topics/impala_max_errors.xml"/>
+          <topicref href="topics/impala_max_io_buffers.xml"/>
+          <topicref href="topics/impala_max_scan_range_length.xml"/>
+          <topicref href="topics/impala_mem_limit.xml"/>
+          <topicref href="topics/impala_num_nodes.xml"/>
+          <topicref href="topics/impala_num_scanner_threads.xml"/>
+          <topicref href="topics/impala_parquet_compression_codec.xml"/>
+          <topicref href="topics/impala_parquet_file_size.xml"/>
+          <topicref href="topics/impala_query_timeout_s.xml"/>
+          <topicref href="topics/impala_request_pool.xml"/>
+          <topicref href="topics/impala_reservation_request_timeout.xml"/>
+          <topicref href="topics/impala_support_start_over.xml"/>
+          <topicref href="topics/impala_sync_ddl.xml"/>
+          <topicref href="topics/impala_v_cpu_cores.xml"/>
+        </topicref>
+      <topicref href="topics/impala_show.xml"/>
+      <topicref href="topics/impala_truncate_table.xml"/>
+      <topicref audience="impala_next" href="topics/impala_update.xml"/>
+      <topicref href="topics/impala_use.xml"/>
+    </topicref>
+    <topicref href="topics/impala_functions.xml">
+      <topicref href="topics/impala_math_functions.xml"/>
+      <topicref href="topics/impala_bit_functions.xml"/>
+      <topicref href="topics/impala_conversion_functions.xml"/>
+      <topicref href="topics/impala_datetime_functions.xml"/>
+      <topicref href="topics/impala_conditional_functions.xml"/>
+      <topicref href="topics/impala_string_functions.xml"/>
+      <topicref href="topics/impala_misc_functions.xml"/>
+      <topicref href="topics/impala_aggregate_functions.xml">
+        <topicref href="topics/impala_appx_median.xml"/>
+        <topicref href="topics/impala_avg.xml"/>
+        <topicref href="topics/impala_count.xml"/>
+        <topicref href="topics/impala_group_concat.xml"/>
+        <topicref href="topics/impala_max.xml"/>
+        <topicref href="topics/impala_min.xml"/>
+        <topicref href="topics/impala_ndv.xml"/>
+        <topicref href="topics/impala_stddev.xml"/>
+        <topicref href="topics/impala_sum.xml"/>
+        <topicref href="topics/impala_variance.xml"/>
+      </topicref>
+      <topicref href="topics/impala_analytic_functions.xml"/>
+      <topicref href="topics/impala_udf.xml"/>
+    </topicref>
+    <topicref href="topics/impala_langref_unsupported.xml"/>
+    <topicref href="topics/impala_porting.xml"/>
+</map>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/shared/ImpalaVariables.xml
----------------------------------------------------------------------
diff --git a/docs/shared/ImpalaVariables.xml b/docs/shared/ImpalaVariables.xml
new file mode 100644
index 0000000..164bbb2
--- /dev/null
+++ b/docs/shared/ImpalaVariables.xml
@@ -0,0 +1,52 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="impala_vars">
+  <title>Cloudera Impala Variables</title>
+  <prolog id="prolog_slg_nmv_km">
+    <metadata id="metadata_ecq_qmv_km">
+      <prodinfo audience="PDF" id="prodinfo_for_html">
+        <prodname>Impala</prodname>
+        <vrmlist>
+          <vrm version="Impala 2.7.x / CDH 5.9.x"/>
+        </vrmlist>
+      </prodinfo>
+      <prodinfo audience="HTML" id="prodinfo_for_pdf">
+        <prodname></prodname>
+        <vrmlist>
+          <vrm version="Impala 2.7.x / CDH 5.9.x"/>
+        </vrmlist>
+      </prodinfo>
+    </metadata>
+  </prolog>
+  <conbody>
+    <p>Substitution variables for denoting features available in release X or higher.
+       The upstream docs can refer to the Impala release number.
+       The docs included with a distro can refer to the distro release number by
+       editing the values here.
+       <ul>
+        <li><ph id="impala27">CDH 5.9</ph></li>
+        <li><ph id="impala26">CDH 5.8</ph></li>
+        <li><ph id="impala25">CDH 5.7</ph></li>
+        <li><ph id="impala24">CDH 5.6</ph></li>
+        <li><ph id="impala23">CDH 5.5</ph></li>
+        <li><ph id="impala22">CDH 5.4</ph></li>
+        <li><ph id="impala21">CDH 5.3</ph></li>
+        <li><ph id="impala20">CDH 5.2</ph></li>
+        <li><ph id="impala14">CDH 5.1</ph></li>
+        <li><ph id="impala13">CDH 5.0</ph></li>
+      </ul>
+    </p>
+    <p>Release Version Variable - <ph id="ReleaseVersion">Impala 2.7.x / CDH 5.9.x</ph></p>
+    <p>Banner for examples showing shell version -<ph id="ShellBanner">(Shell
+      build version: Impala Shell v2.7.x (<varname>hash</varname>) built on
+      <varname>date</varname>)</ph></p>
+    <p>Banner for examples showing impalad version -<ph id="ImpaladBanner">Server version: impalad version 2.7.x (build
+      x.y.z)</ph></p>
+    <data name="version-message" id="version-message">
+      <foreign>
+        <lines xml:space="preserve">This is the documentation for <data name="version"/>.
+Documentation for other versions is available at <xref href="http://www.cloudera.com/content/support/en/documentation.html" scope="external" format="html">Cloudera Documentation</xref>.</lines>
+      </foreign>
+    </data>
+  </conbody>
+</concept>


[23/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_joins.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_joins.xml b/docs/topics/impala_joins.xml
new file mode 100644
index 0000000..ddd2f2b
--- /dev/null
+++ b/docs/topics/impala_joins.xml
@@ -0,0 +1,534 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="joins">
+
+  <title>Joins in Impala SELECT Statements</title>
+  <titlealts audience="PDF"><navtitle>Joins</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">joins</indexterm>
+      A join query is a <codeph>SELECT</codeph> statement that combines data from two or more tables,
+      and returns a result set containing items from some or all of those tables. It is a way to
+      cross-reference and correlate related data that is organized into multiple tables, typically
+      using identifiers that are repeated in each of the joined tables.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/join_types"/>
+
+<codeblock>SELECT <varname>select_list</varname> FROM
+  <varname>table_or_subquery1</varname> [INNER] JOIN <varname>table_or_subquery2</varname> |
+  <varname>table_or_subquery1</varname> {LEFT [OUTER] | RIGHT [OUTER] | FULL [OUTER]} JOIN <varname>table_or_subquery2</varname> |
+  <varname>table_or_subquery1</varname> {LEFT | RIGHT} SEMI JOIN <varname>table_or_subquery2</varname> |
+  <ph rev="2.0.0"><varname>table_or_subquery1</varname> {LEFT | RIGHT} ANTI JOIN <varname>table_or_subquery2</varname> |</ph>
+    [ ON <varname>col1</varname> = <varname>col2</varname> [AND <varname>col3</varname> = <varname>col4</varname> ...] |
+      USING (<varname>col1</varname> [, <varname>col2</varname> ...]) ]
+  [<varname>other_join_clause</varname> ...]
+[ WHERE <varname>where_clauses</varname> ]
+
+SELECT <varname>select_list</varname> FROM
+  <varname>table_or_subquery1</varname>, <varname>table_or_subquery2</varname> [, <varname>table_or_subquery3</varname> ...]
+  [<varname>other_join_clause</varname> ...]
+WHERE
+    <varname>col1</varname> = <varname>col2</varname> [AND <varname>col3</varname> = <varname>col4</varname> ...]
+
+SELECT <varname>select_list</varname> FROM
+  <varname>table_or_subquery1</varname> CROSS JOIN <varname>table_or_subquery2</varname>
+  [<varname>other_join_clause</varname> ...]
+[ WHERE <varname>where_clauses</varname> ]</codeblock>
+
+    <p>
+      <b>SQL-92 and SQL-89 Joins:</b>
+    </p>
+
+    <p>
+      Queries with the explicit <codeph>JOIN</codeph> keywords are known as SQL-92 style joins, referring to the
+      level of the SQL standard where they were introduced. The corresponding <codeph>ON</codeph> or
+      <codeph>USING</codeph> clauses clearly show which columns are used as the join keys in each case:
+    </p>
+
+<codeblock>SELECT t1.c1, t2.c2 FROM <b>t1 JOIN t2</b>
+  <b>ON t1.id = t2.id and t1.type_flag = t2.type_flag</b>
+  WHERE t1.c1 &gt; 100;
+
+SELECT t1.c1, t2.c2 FROM <b>t1 JOIN t2</b>
+  <b>USING (id, type_flag)</b>
+  WHERE t1.c1 &gt; 100;</codeblock>
+
+    <p>
+      The <codeph>ON</codeph> clause is a general way to compare columns across the two tables, even if the column
+      names are different. The <codeph>USING</codeph> clause is a shorthand notation for specifying the join
+      columns, when the column names are the same in both tables. You can code equivalent <codeph>WHERE</codeph>
+      clauses that compare the columns, instead of <codeph>ON</codeph> or <codeph>USING</codeph> clauses, but that
+      practice is not recommended because mixing the join comparisons with other filtering clauses is typically
+      less readable and harder to maintain.
+    </p>
+
+    <p>
+      Queries with a comma-separated list of tables and subqueries are known as SQL-89 style joins. In these
+      queries, the equality comparisons between columns of the joined tables go in the <codeph>WHERE</codeph>
+      clause alongside other kinds of comparisons. This syntax is easy to learn, but it is also easy to
+      accidentally remove a <codeph>WHERE</codeph> clause needed for the join to work correctly.
+    </p>
+
+<codeblock>SELECT t1.c1, t2.c2 FROM <b>t1, t2</b>
+  WHERE
+  <b>t1.id = t2.id AND t1.type_flag = t2.type_flag</b>
+  AND t1.c1 &gt; 100;</codeblock>
+
+    <p>
+      <b>Self-joins:</b>
+    </p>
+
+    <p>
+      Impala can do self-joins, for example to join on two different columns in the same table to represent
+      parent-child relationships or other tree-structured data. There is no explicit syntax for this; just use the
+      same table name for both the left-hand and right-hand table, and assign different table aliases to use when
+      referring to the fully qualified column names:
+    </p>
+
+<codeblock>-- Combine fields from both parent and child rows.
+SELECT lhs.id, rhs.parent, lhs.c1, rhs.c2 FROM tree_data lhs, tree_data rhs WHERE lhs.id = rhs.parent;</codeblock>
+
+    <p>
+      <b>Cartesian joins:</b>
+    </p>
+
+    <p>
+      To avoid producing huge result sets by mistake, Impala does not allow Cartesian joins of the form:
+<codeblock>SELECT ... FROM t1 JOIN t2;
+SELECT ... FROM t1, t2;</codeblock>
+      If you intend to join the tables based on common values, add <codeph>ON</codeph> or <codeph>WHERE</codeph>
+      clauses to compare columns across the tables. If you truly intend to do a Cartesian join, use the
+      <codeph>CROSS JOIN</codeph> keyword as the join operator. The <codeph>CROSS JOIN</codeph> form does not use
+      any <codeph>ON</codeph> clause, because it produces a result set with all combinations of rows from the
+      left-hand and right-hand tables. The result set can still be filtered by subsequent <codeph>WHERE</codeph>
+      clauses. For example:
+    </p>
+
+<codeblock>SELECT ... FROM t1 CROSS JOIN t2;
+SELECT ... FROM t1 CROSS JOIN t2 WHERE <varname>tests_on_non_join_columns</varname>;</codeblock>
+
+    <p>
+      <b>Inner and outer joins:</b>
+    </p>
+
+    <p>
+      An inner join is the most common and familiar type: rows in the result set contain the requested columns from
+      the appropriate tables, for all combinations of rows where the join columns of the tables have identical
+      values. If a column with the same name occurs in both tables, use a fully qualified name or a column alias to
+      refer to the column in the select list or other clauses. Impala performs inner joins by default for both
+      SQL-89 and SQL-92 join syntax:
+    </p>
+
+<codeblock>-- The following 3 forms are all equivalent.
+SELECT t1.id, c1, c2 FROM t1, t2 WHERE t1.id = t2.id;
+SELECT t1.id, c1, c2 FROM t1 JOIN t2 ON t1.id = t2.id;
+SELECT t1.id, c1, c2 FROM t1 INNER JOIN t2 ON t1.id = t2.id;</codeblock>
+
+    <p>
+      An outer join retrieves all rows from the left-hand table, or the right-hand table, or both; wherever there
+      is no matching data in the table on the other side of the join, the corresponding columns in the result set
+      are set to <codeph>NULL</codeph>. To perform an outer join, include the <codeph>OUTER</codeph> keyword in the
+      join operator, along with either <codeph>LEFT</codeph>, <codeph>RIGHT</codeph>, or <codeph>FULL</codeph>:
+    </p>
+
+<codeblock>SELECT * FROM t1 LEFT OUTER JOIN t2 ON t1.id = t2.id;
+SELECT * FROM t1 RIGHT OUTER JOIN t2 ON t1.id = t2.id;
+SELECT * FROM t1 FULL OUTER JOIN t2 ON t1.id = t2.id;</codeblock>
+
+    <p>
+      For outer joins, Impala requires SQL-92 syntax; that is, the <codeph>JOIN</codeph> keyword instead of
+      comma-separated table names. Impala does not support vendor extensions such as <codeph>(+)</codeph> or
+      <codeph>*=</codeph> notation for doing outer joins with SQL-89 query syntax.
+    </p>
+
+    <p>
+      <b>Equijoins and Non-Equijoins:</b>
+    </p>
+
+    <p>
+      By default, Impala requires an equality comparison between the left-hand and right-hand tables, either
+      through <codeph>ON</codeph>, <codeph>USING</codeph>, or <codeph>WHERE</codeph> clauses. These types of
+      queries are classified broadly as equijoins. Inner, outer, full, and semi joins can all be equijoins based on
+      the presence of equality tests between columns in the left-hand and right-hand tables.
+    </p>
+
+    <p>
+      In Impala 1.2.2 and higher, non-equijoin queries are also possible, with comparisons such as
+      <codeph>!=</codeph> or <codeph>&lt;</codeph> between the join columns. These kinds of queries require care to
+      avoid producing huge result sets that could exceed resource limits. Once you have planned a non-equijoin
+      query that produces a result set of acceptable size, you can code the query using the <codeph>CROSS
+      JOIN</codeph> operator, and add the extra comparisons in the <codeph>WHERE</codeph> clause:
+    </p>
+
+<codeblock>SELECT * FROM t1 CROSS JOIN t2 WHERE t1.total &gt; t2.maximum_price;</codeblock>
+
+    <p rev="2.3.0">
+      In <keyword keyref="impala23_full"/> and higher, additional non-equijoin queries are possible due to the addition
+      of nested loop joins. These queries typically involve <codeph>SEMI JOIN</codeph>,
+      <codeph>ANTI JOIN</codeph>, or <codeph>FULL OUTER JOIN</codeph> clauses.
+      Impala sometimes also uses nested loop joins internally when evaluating <codeph>OUTER JOIN</codeph>
+      queries involving complex type columns.
+      Query phases involving nested loop joins do not use the spill-to-disk mechanism if they
+      exceed the memory limit. Impala decides internally when to use each join mechanism; you cannot
+      specify any query hint to choose between the nested loop join or the original hash join algorithm.
+    </p>
+
+<codeblock rev="2.3.0">SELECT * FROM t1 LEFT OUTER JOIN t2 ON t1.int_col &lt; t2.int_col;</codeblock>
+
+    <p>
+      <b>Semi-joins:</b>
+    </p>
+
+    <p>
+      Semi-joins are a relatively rarely used variation. With the left semi-join, only data from the left-hand
+      table is returned, for rows where there is matching data in the right-hand table, based on comparisons
+      between join columns in <codeph>ON</codeph> or <codeph>WHERE</codeph> clauses. Only one instance of each row
+      from the left-hand table is returned, regardless of how many matching rows exist in the right-hand table.
+      <ph rev="2.0.0">A right semi-join (available in Impala 2.0 and higher) reverses the comparison and returns
+      data from the right-hand table.</ph>
+    </p>
+
+<codeblock>SELECT t1.c1, t1.c2, t1.c2 FROM t1 LEFT SEMI JOIN t2 ON t1.id = t2.id;</codeblock>
+
+    <p>
+      <b>Natural joins (not supported):</b>
+    </p>
+
+    <p>
+      Impala does not support the <codeph>NATURAL JOIN</codeph> operator, again to avoid inconsistent or huge
+      result sets. Natural joins do away with the <codeph>ON</codeph> and <codeph>USING</codeph> clauses, and
+      instead automatically join on all columns with the same names in the left-hand and right-hand tables. This
+      kind of query is not recommended for rapidly evolving data structures such as are typically used in Hadoop.
+      Thus, Impala does not support the <codeph>NATURAL JOIN</codeph> syntax, which can produce different query
+      results as columns are added to or removed from tables.
+    </p>
+
+    <p>
+      If you do have any queries that use <codeph>NATURAL JOIN</codeph>, make sure to rewrite them with explicit
+      <codeph>USING</codeph> clauses, because Impala could interpret the <codeph>NATURAL</codeph> keyword as a
+      table alias:
+    </p>
+
+<codeblock>-- 'NATURAL' is interpreted as an alias for 't1' and Impala attempts an inner join,
+-- resulting in an error because inner joins require explicit comparisons between columns.
+SELECT t1.c1, t2.c2 FROM t1 NATURAL JOIN t2;
+ERROR: NotImplementedException: Join with 't2' requires at least one conjunctive equality predicate.
+  To perform a Cartesian product between two tables, use a CROSS JOIN.
+
+-- If you expect the tables to have identically named columns with matching values,
+-- list the corresponding column names in a USING clause.
+SELECT t1.c1, t2.c2 FROM t1 JOIN t2 USING (id, type_flag, name, address);</codeblock>
+
+    <p rev="2.0.0">
+      <b>Anti-joins (<keyword keyref="impala20_full"/> and higher only):</b>
+    </p>
+
+    <p rev="2.0.0">
+      Impala supports the <codeph>LEFT ANTI JOIN</codeph> and <codeph>RIGHT ANTI JOIN</codeph> clauses in
+      <keyword keyref="impala20"/> and higher. The <codeph>LEFT</codeph> or <codeph>RIGHT</codeph>
+      keyword is required for this kind of join. For <codeph>LEFT ANTI JOIN</codeph>, this clause returns those
+      values from the left-hand table that have no matching value in the right-hand table. <codeph>RIGHT ANTI
+      JOIN</codeph> reverses the comparison and returns values from the right-hand table. You can express this
+      negative relationship either through the <codeph>ANTI JOIN</codeph> clause or through a <codeph>NOT
+      EXISTS</codeph> operator with a subquery.
+    </p>
+
+<!-- Restriction lifted in Impala 2.0.
+<p>
+Impala does not support <codeph>WHERE</codeph> clauses
+such as <codeph>IN (<varname>subquery</varname>)</codeph>,
+<codeph>NOT IN (<varname>subquery</varname>)</codeph>,
+<codeph>EXISTS (<varname>subquery</varname>)</codeph>,
+and <codeph>NOT EXISTS (<varname>subquery</varname>)</codeph>.
+Therefore from a practical standpoint, you cannot
+express an anti-join condition, where values from one table
+are returned only if no matching values are present in another table.
+</p>
+-->
+
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+<!-- To do: reuse some complex types examples with joins here or under Examples farther down. -->
+
+    <p rev="2.3.0">
+      When referring to a column with a complex type (<codeph>STRUCT</codeph>, <codeph>ARRAY</codeph>, or <codeph>MAP</codeph>)
+      in a query, you use join notation to <q>unpack</q> the scalar fields of the struct, the elements of the array, or
+      the key-value pairs of the map. (The join notation is not required for aggregation operations, such as
+      <codeph>COUNT()</codeph> or <codeph>SUM()</codeph> for array elements.) Because Impala recognizes which complex type elements are associated with which row
+      of the result set, you use the same syntax as for a cross or cartesian join, without an explicit join condition.
+      See <xref href="impala_complex_types.xml#complex_types"/> for details about Impala support for complex types.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      You typically use join queries in situations like these:
+    </p>
+
+    <ul>
+      <li>
+        When related data arrives from different sources, with each data set physically residing in a separate
+        table. For example, you might have address data from business records that you cross-check against phone
+        listings or census data.
+        <note>
+          Impala can join tables of different file formats, including Impala-managed tables and HBase tables. For
+          example, you might keep small dimension tables in HBase, for convenience of single-row lookups and
+          updates, and for the larger fact tables use Parquet or other binary file format optimized for scan
+          operations. Then, you can issue a join query to cross-reference the fact tables with the dimension
+          tables.
+        </note>
+      </li>
+
+      <li>
+        When data is normalized, a technique for reducing data duplication by dividing it across multiple tables.
+        This kind of organization is often found in data that comes from traditional relational database systems.
+        For example, instead of repeating some long string such as a customer name in multiple tables, each table
+        might contain a numeric customer ID. Queries that need to display the customer name could <q>join</q> the
+        table that specifies which customer ID corresponds to which name.
+      </li>
+
+      <li>
+        When certain columns are rarely needed for queries, so they are moved into separate tables to reduce
+        overhead for common queries. For example, a <codeph>biography</codeph> field might be rarely needed in
+        queries on employee data. Putting that field in a separate table reduces the amount of I/O for common
+        queries on employee addresses or phone numbers. Queries that do need the <codeph>biography</codeph> column
+        can retrieve it by performing a join with that separate table.
+      </li>
+
+      <li>
+        In <keyword keyref="impala23_full"/> or higher, when referring to complex type columns in queries.
+        See <xref href="impala_complex_types.xml#complex_types"/> for details.
+      </li>
+    </ul>
+
+    <p>
+      When comparing columns with the same names in <codeph>ON</codeph> or <codeph>WHERE</codeph> clauses, use the
+      fully qualified names such as <codeph><varname>db_name</varname>.<varname>table_name</varname></codeph>, or
+      assign table aliases, column aliases, or both to make the code more compact and understandable:
+    </p>
+
+<codeblock>select t1.c1 as first_id, t2.c2 as second_id from
+  t1 join t2 on first_id = second_id;
+
+select fact.custno, dimension.custno from
+  customer_data as fact join customer_address as dimension
+  using (custno)</codeblock>
+
+    <note>
+      <p>
+        Performance for join queries is a crucial aspect for Impala, because complex join queries are
+        resource-intensive operations. An efficient join query produces much less network traffic and CPU overhead
+        than an inefficient one. For best results:
+      </p>
+      <ul>
+        <li rev="1.2">
+          Make sure that both <xref href="impala_perf_stats.xml#perf_stats">table and column statistics</xref> are
+          available for all the tables involved in a join query, and especially for the columns referenced in any
+          join conditions. Impala uses the statistics to automatically deduce an efficient join order.
+          Use <xref href="impala_show.xml#show"><codeph>SHOW TABLE STATS <varname>table_name</varname></codeph> and
+          <codeph>SHOW COLUMN STATS <varname>table_name</varname></codeph></xref> to check if statistics are
+          already present. Issue the <codeph>COMPUTE STATS <varname>table_name</varname></codeph> for a nonpartitioned table,
+          or (in Impala 2.1.0 and higher) <codeph>COMPUTE INCREMENTAL STATS <varname>table_name</varname></codeph>
+          for a partitioned table, to collect the initial statistics at both the table and column levels, and to keep the
+          statistics up to date after any substantial <codeph>INSERT</codeph> or <codeph>LOAD DATA</codeph> operations.
+        </li>
+
+        <li rev="1.2">
+          If table or column statistics are not available, join the largest table first. You can check the
+          existence of statistics with the <codeph>SHOW TABLE STATS <varname>table_name</varname></codeph> and
+          <codeph>SHOW COLUMN STATS <varname>table_name</varname></codeph> statements.
+        </li>
+
+        <li rev="1.2.2">
+          If table or column statistics are not available, join subsequent tables according to which table has the
+          most selective filter, based on overall size and <codeph>WHERE</codeph> clauses. Joining the table with
+          the most selective filter results in the fewest number of rows being returned.
+        </li>
+      </ul>
+      <p>
+        For more information and examples of performance for join queries, see
+        <xref href="impala_perf_joins.xml#perf_joins"/>.
+      </p>
+    </note>
+
+    <p>
+      To control the result set from a join query, include the names of corresponding column names in both tables
+      in an <codeph>ON</codeph> or <codeph>USING</codeph> clause, or by coding equality comparisons for those
+      columns in the <codeph>WHERE</codeph> clause.
+    </p>
+
+<codeblock>[localhost:21000] &gt; select c_last_name, ca_city from customer join customer_address where c_customer_sk = ca_address_sk;
++-------------+-----------------+
+| c_last_name | ca_city         |
++-------------+-----------------+
+| Lewis       | Fairfield       |
+| Moses       | Fairview        |
+| Hamilton    | Pleasant Valley |
+| White       | Oak Ridge       |
+| Moran       | Glendale        |
+...
+| Richards    | Lakewood         |
+| Day         | Lebanon          |
+| Painter     | Oak Hill         |
+| Bentley     | Greenfield       |
+| Jones       | Stringtown       |
++-------------+------------------+
+Returned 50000 row(s) in 9.82s</codeblock>
+
+    <p>
+      One potential downside of joins is the possibility of excess resource usage in poorly constructed queries.
+      Impala imposes restrictions on join queries to guard against such issues. To minimize the chance of runaway
+      queries on large data sets, Impala requires every join query to contain at least one equality predicate
+      between the columns of the various tables. For example, if <codeph>T1</codeph> contains 1000 rows and
+      <codeph>T2</codeph> contains 1,000,000 rows, a query <codeph>SELECT <varname>columns</varname> FROM t1 JOIN
+      t2</codeph> could return up to 1 billion rows (1000 * 1,000,000); Impala requires that the query include a
+      clause such as <codeph>ON t1.c1 = t2.c2</codeph> or <codeph>WHERE t1.c1 = t2.c2</codeph>.
+    </p>
+
+    <p>
+      Because even with equality clauses, the result set can still be large, as we saw in the previous example, you
+      might use a <codeph>LIMIT</codeph> clause to return a subset of the results:
+    </p>
+
+<codeblock>[localhost:21000] &gt; select c_last_name, ca_city from customer, customer_address where c_customer_sk = ca_address_sk limit 10;
++-------------+-----------------+
+| c_last_name | ca_city         |
++-------------+-----------------+
+| Lewis       | Fairfield       |
+| Moses       | Fairview        |
+| Hamilton    | Pleasant Valley |
+| White       | Oak Ridge       |
+| Moran       | Glendale        |
+| Sharp       | Lakeview        |
+| Wiles       | Farmington      |
+| Shipman     | Union           |
+| Gilbert     | New Hope        |
+| Brunson     | Martinsville    |
++-------------+-----------------+
+Returned 10 row(s) in 0.63s</codeblock>
+
+    <p>
+      Or you might use additional comparison operators or aggregation functions to condense a large result set into
+      a smaller set of values:
+    </p>
+
+<codeblock>[localhost:21000] &gt; -- Find the names of customers who live in one particular town.
+[localhost:21000] &gt; select distinct c_last_name from customer, customer_address where
+  c_customer_sk = ca_address_sk
+  and ca_city = "Green Acres";
++---------------+
+| c_last_name   |
++---------------+
+| Hensley       |
+| Pearson       |
+| Mayer         |
+| Montgomery    |
+| Ricks         |
+...
+| Barrett       |
+| Price         |
+| Hill          |
+| Hansen        |
+| Meeks         |
++---------------+
+Returned 332 row(s) in 0.97s
+
+[localhost:21000] &gt; -- See how many different customers in this town have names starting with "A".
+[localhost:21000] &gt; select count(distinct c_last_name) from customer, customer_address where
+  c_customer_sk = ca_address_sk
+  and ca_city = "Green Acres"
+  and substr(c_last_name,1,1) = "A";
++-----------------------------+
+| count(distinct c_last_name) |
++-----------------------------+
+| 12                          |
++-----------------------------+
+Returned 1 row(s) in 1.00s</codeblock>
+
+    <p>
+      Because a join query can involve reading large amounts of data from disk, sending large amounts of data
+      across the network, and loading large amounts of data into memory to do the comparisons and filtering, you
+      might do benchmarking, performance analysis, and query tuning to find the most efficient join queries for
+      your data set, hardware capacity, network configuration, and cluster workload.
+    </p>
+
+    <p>
+      The two categories of joins in Impala are known as <b>partitioned joins</b> and <b>broadcast joins</b>. If
+      inaccurate table or column statistics, or some quirk of the data distribution, causes Impala to choose the
+      wrong mechanism for a particular join, consider using query hints as a temporary workaround. For details, see
+      <xref href="impala_hints.xml#hints"/>.
+    </p>
+
+    <p rev="2.5.0">
+      <b>Handling NULLs in Join Columns:</b>
+    </p>
+
+    <p rev="2.5.0">
+      By default, join key columns do not match if either one contains a <codeph>NULL</codeph> value.
+      To treat such columns as equal if both contain <codeph>NULL</codeph>, you can use an expression
+      such as <codeph>A = B OR (A IS NULL AND B IS NULL)</codeph>.
+      In <keyword keyref="impala25_full"/> and higher, the <codeph>&lt;=&gt;</codeph> operator (shorthand for
+      <codeph>IS NOT DISTINCT FROM</codeph>) performs the same comparison in a concise and efficient form.
+      The <codeph>&lt;=&gt;</codeph> operator is more efficient in for comparing join keys in a <codeph>NULL</codeph>-safe
+      manner, because the operator can use a hash join while the <codeph>OR</codeph> expression cannot.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      The following examples refer to these simple tables containing small sets of integers:
+<codeblock>[localhost:21000] &gt; create table t1 (x int);
+[localhost:21000] &gt; insert into t1 values (1), (2), (3), (4), (5), (6);
+
+[localhost:21000] &gt; create table t2 (y int);
+[localhost:21000] &gt; insert into t2 values (2), (4), (6);
+
+[localhost:21000] &gt; create table t3 (z int);
+[localhost:21000] &gt; insert into t3 values (1), (3), (5);
+</codeblock>
+    </p>
+
+<!-- To do: fill in examples for other join types. -->
+
+    <p>
+      The following example demonstrates an anti-join, returning the values from <codeph>T1</codeph> that do not
+      exist in <codeph>T2</codeph> (in this case, the odd numbers 1, 3, and 5):
+    </p>
+
+<codeblock>[localhost:21000] &gt; select x from t1 left anti join t2 on (t1.x = t2.y);
++---+
+| x |
++---+
+| 1 |
+| 3 |
+| 5 |
++---+
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      See these tutorials for examples of different kinds of joins:
+    </p>
+
+    <ul>
+      <li>
+        <xref href="impala_tutorial.xml#tut_cross_join"/>
+      </li>
+    </ul>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_kerberos.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_kerberos.xml b/docs/topics/impala_kerberos.xml
new file mode 100644
index 0000000..6393580
--- /dev/null
+++ b/docs/topics/impala_kerberos.xml
@@ -0,0 +1,363 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="kerberos">
+
+  <title>Enabling Kerberos Authentication for Impala</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="Kerberos"/>
+      <data name="Category" value="Authentication"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Configuring"/>
+      <data name="Category" value="Starting and Stopping"/>
+      <data name="Category" value="Administrators"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Impala supports Kerberos authentication. For more information on enabling Kerberos authentication, see the
+      topic on Configuring Hadoop Security in the
+      <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_cdh5_hadoop_security.html" scope="external" format="html">CDH 5 Security Guide</xref>.
+    </p>
+
+    <p>
+      When using Impala in a managed environment, Cloudera Manager automatically completes Kerberos configuration.
+      In an unmanaged environment, create a Kerberos principal for each host running <cmdname>impalad</cmdname> or
+      <cmdname>statestored</cmdname>. <ph rev="upstream">Cloudera</ph> recommends using a consistent format, such as
+      <codeph>impala/_HOST@Your-Realm</codeph>, but you can use any three-part Kerberos server principal.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/user_kerberized"/>
+
+    <note conref="../shared/impala_common.xml#common/authentication_vs_authorization"/>
+
+    <p outputclass="toc inpage"/>
+
+    <p>
+      An alternative form of authentication you can use is LDAP, described in <xref href="impala_ldap.xml#ldap"/>.
+    </p>
+  </conbody>
+
+  <concept id="kerberos_prereqs">
+
+    <title>Requirements for Using Impala with Kerberos</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Requirements"/>
+      <data name="Category" value="Planning"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p conref="../shared/impala_common.xml#common/rhel5_kerberos"/>
+
+<!-- This note adapted from the one at http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Security-Guide/cdh4sg_topic_3_4.html.
+   Ideally should be conref'ed in both places. -->
+
+      <note type="important">
+        <p>
+          If you plan to use Impala in your cluster, you must configure your KDC to allow tickets to be renewed,
+          and you must configure <filepath>krb5.conf</filepath> to request renewable tickets. Typically, you can do
+          this by adding the <codeph>max_renewable_life</codeph> setting to your realm in
+          <filepath>kdc.conf</filepath>, and by adding the <filepath>renew_lifetime</filepath> parameter to the
+          <filepath>libdefaults</filepath> section of <filepath>krb5.conf</filepath>. For more information about
+          renewable tickets, see the
+          <xref href="http://web.mit.edu/Kerberos/krb5-1.8/" scope="external" format="html"> Kerberos
+          documentation</xref>.
+        </p>
+        <p rev="1.2">
+          Currently, you cannot use the resource management feature in CDH 5 on a cluster that has Kerberos
+          authentication enabled.
+        </p>
+      </note>
+
+      <p>
+        Start all <cmdname>impalad</cmdname> and <cmdname>statestored</cmdname> daemons with the
+        <codeph>--principal</codeph> and <codeph>--keytab-file</codeph> flags set to the principal and full path
+        name of the <codeph>keytab</codeph> file containing the credentials for the principal.
+      </p>
+
+      <p>
+        Impala supports the Cloudera ODBC driver and the Kerberos interface provided. To use Kerberos through the
+        ODBC driver, the host type must be set depending on the level of the ODBD driver:
+      </p>
+
+      <ul>
+        <li>
+          <codeph>SecImpala</codeph> for the ODBC 1.0 driver.
+        </li>
+
+        <li>
+          <codeph>SecBeeswax</codeph> for the ODBC 1.2 driver.
+        </li>
+
+        <li>
+          Blank for the ODBC 2.0 driver or higher, when connecting to a secure cluster.
+        </li>
+
+        <li>
+          <codeph>HS2NoSasl</codeph> for the ODBC 2.0 driver or higher, when connecting to a non-secure cluster.
+        </li>
+      </ul>
+
+      <p>
+        To enable Kerberos in the Impala shell, start the <cmdname>impala-shell</cmdname> command using the
+        <codeph>-k</codeph> flag.
+      </p>
+
+      <p>
+        To enable Impala to work with Kerberos security on your Hadoop cluster, make sure you perform the
+        installation and configuration steps in
+        <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/sg_authentication.html" scope="external" format="html">Authentication in the CDH 5 Security Guide</xref>.
+        Also note that when Kerberos security is enabled in Impala, a web browser that
+        supports Kerberos HTTP SPNEGO is required to access the Impala web console (for example, Firefox, Internet
+        Explorer, or Chrome).
+      </p>
+
+      <p>
+        If the NameNode, Secondary NameNode, DataNode, JobTracker, TaskTrackers, ResourceManager, NodeManagers,
+        HttpFS, Oozie, Impala, or Impala statestore services are configured to use Kerberos HTTP SPNEGO
+        authentication, and two or more of these services are running on the same host, then all of the running
+        services must use the same HTTP principal and keytab file used for their HTTP endpoints.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="kerberos_config">
+
+    <title>Configuring Impala to Support Kerberos Security</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Configuring"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        Enabling Kerberos authentication for Impala involves steps that can be summarized as follows:
+      </p>
+
+      <ul>
+        <li>
+          Creating service principals for Impala and the HTTP service. Principal names take the form:
+          <codeph><varname>serviceName</varname>/<varname>fully.qualified.domain.name</varname>@<varname>KERBEROS.REALM</varname></codeph>
+        </li>
+
+        <li>
+          Creating, merging, and distributing key tab files for these principals.
+        </li>
+
+        <li>
+          Editing <codeph>/etc/default/impala</codeph> (in cluster not managed by Cloudera Manager), or editing the
+          <uicontrol>Security</uicontrol> settings in the Cloudera Manager interface, to accommodate Kerberos
+          authentication.
+        </li>
+      </ul>
+    </conbody>
+
+    <concept id="kerberos_setup">
+
+      <title>Enabling Kerberos for Impala</title>
+
+      <conbody>
+
+<!--
+      <p>
+        <b>To enable Kerberos for Impala:</b>
+      </p>
+-->
+
+        <ol>
+          <li>
+            Create an Impala service principal, specifying the name of the OS user that the Impala daemons run
+            under, the fully qualified domain name of each node running <cmdname>impalad</cmdname>, and the realm
+            name. For example:
+<codeblock>$ kadmin
+kadmin: addprinc -requires_preauth -randkey impala/impala_host.example.com@TEST.EXAMPLE.COM</codeblock>
+          </li>
+
+          <li>
+            Create an HTTP service principal. For example:
+<codeblock>kadmin: addprinc -randkey HTTP/impala_host.example.com@TEST.EXAMPLE.COM</codeblock>
+            <note>
+              The <codeph>HTTP</codeph> component of the service principal must be uppercase as shown in the
+              preceding example.
+            </note>
+          </li>
+
+          <li>
+            Create <codeph>keytab</codeph> files with both principals. For example:
+<codeblock>kadmin: xst -k impala.keytab impala/impala_host.example.com
+kadmin: xst -k http.keytab HTTP/impala_host.example.com
+kadmin: quit</codeblock>
+          </li>
+
+          <li>
+            Use <codeph>ktutil</codeph> to read the contents of the two keytab files and then write those contents
+            to a new file. For example:
+<codeblock>$ ktutil
+ktutil: rkt impala.keytab
+ktutil: rkt http.keytab
+ktutil: wkt impala-http.keytab
+ktutil: quit</codeblock>
+          </li>
+
+          <li>
+            (Optional) Test that credentials in the merged keytab file are valid, and that the <q>renew until</q>
+            date is in the future. For example:
+<codeblock>$ klist -e -k -t impala-http.keytab</codeblock>
+          </li>
+
+          <li>
+            Copy the <filepath>impala-http.keytab</filepath> file to the Impala configuration directory. Change the
+            permissions to be only read for the file owner and change the file owner to the <codeph>impala</codeph>
+            user. By default, the Impala user and group are both named <codeph>impala</codeph>. For example:
+<codeblock>$ cp impala-http.keytab /etc/impala/conf
+$ cd /etc/impala/conf
+$ chmod 400 impala-http.keytab
+$ chown impala:impala impala-http.keytab</codeblock>
+          </li>
+
+          <li>
+            Add Kerberos options to the Impala defaults file, <filepath>/etc/default/impala</filepath>. Add the
+            options for both the <cmdname>impalad</cmdname> and <cmdname>statestored</cmdname> daemons, using the
+            <codeph>IMPALA_SERVER_ARGS</codeph> and <codeph>IMPALA_STATE_STORE_ARGS</codeph> variables. For
+            example, you might add:
+<!-- Found these in a discussion post somewhere but not applicable as Impala startup options.
+-kerberos_ticket_life=36000
+-maxrenewlife 7days
+-->
+<codeblock>-kerberos_reinit_interval=60
+-principal=impala_1/impala_host.example.com@TEST.EXAMPLE.COM
+-keytab_file=/var/run/cloudera-scm-agent/process/3212-impala-IMPALAD/impala.keytab</codeblock>
+            <p>
+              For more information on changing the Impala defaults specified in
+              <filepath>/etc/default/impala</filepath>, see
+              <xref href="impala_config_options.xml#config_options">Modifying Impala Startup
+              Options</xref>.
+            </p>
+          </li>
+        </ol>
+
+        <note>
+          Restart <cmdname>impalad</cmdname> and <cmdname>statestored</cmdname> for these configuration changes to
+          take effect.
+        </note>
+      </conbody>
+    </concept>
+  </concept>
+
+  <concept id="kerberos_proxy">
+
+    <title>Enabling Kerberos for Impala with a Proxy Server</title>
+
+    <conbody>
+
+      <p>
+        A common configuration for Impala with High Availability is to use a proxy server to submit requests to the
+        actual <cmdname>impalad</cmdname> daemons on different hosts in the cluster. This configuration avoids
+        connection problems in case of machine failure, because the proxy server can route new requests through one
+        of the remaining hosts in the cluster. This configuration also helps with load balancing, because the
+        additional overhead of being the <q>coordinator node</q> for each query is spread across multiple hosts.
+      </p>
+
+      <p>
+        Although you can set up a proxy server with or without Kerberos authentication, typically users set up a
+        secure Kerberized configuration. For information about setting up a proxy server for Impala, including
+        Kerberos-specific steps, see <xref href="impala_proxy.xml#proxy"/>.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="spnego">
+
+    <title>Using a Web Browser to Access a URL Protected by Kerberos HTTP SPNEGO</title>
+
+    <conbody>
+
+      <p>
+        Your web browser must support Kerberos HTTP SPNEGO. For example, Chrome, Firefox, or Internet Explorer.
+      </p>
+
+      <p>
+        <b>To configure Firefox to access a URL protected by Kerberos HTTP SPNEGO:</b>
+      </p>
+
+      <ol>
+        <li>
+          Open the advanced settings Firefox configuration page by loading the <codeph>about:config</codeph> page.
+        </li>
+
+        <li>
+          Use the <b>Filter</b> text box to find <codeph>network.negotiate-auth.trusted-uris</codeph>.
+        </li>
+
+        <li>
+          Double-click the <codeph>network.negotiate-auth.trusted-uris</codeph> preference and enter the hostname
+          or the domain of the web server that is protected by Kerberos HTTP SPNEGO. Separate multiple domains and
+          hostnames with a comma.
+        </li>
+
+        <li>
+          Click <b>OK</b>.
+        </li>
+      </ol>
+    </conbody>
+  </concept>
+
+  <concept id="kerberos_delegation">
+    <title>Enabling Impala Delegation for Kerberos Users</title>
+    <conbody>
+      <p>
+        See <xref href="impala_delegation.xml#delegation"/> for details about the delegation feature
+        that lets certain users submit queries using the credentials of other users.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="ssl_jdbc_odbc">
+    <title>Using TLS/SSL with Business Intelligence Tools</title>
+    <conbody>
+      <p>
+        You can use Kerberos authentication, TLS/SSL encryption, or both to secure
+        connections from JDBC and ODBC applications to Impala.
+        See <xref href="impala_jdbc.xml#impala_jdbc"/> and <xref href="impala_odbc.xml#impala_odbc"/>
+        for details.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/hive_jdbc_ssl_kerberos_caveat"/>
+    </conbody>
+  </concept>
+
+  <concept id="whitelisting_internal_apis">
+  <title>Enabling Access to Internal Impala APIs for Kerberos Users</title>
+    <conbody>
+    <!-- Reusing (most of) the text from the New Features bullet here. Turn into a conref in both places. -->
+      <p rev="IMPALA-3095">
+        For applications that need direct access
+        to Impala APIs, without going through the HiveServer2 or Beeswax interfaces, you can
+        specify a list of Kerberos users who are allowed to call those APIs. By default, the
+        <codeph>impala</codeph> and <codeph>hdfs</codeph> users are the only ones authorized
+        for this kind of access.
+        Any users not explicitly authorized through the <codeph>internal_principals_whitelist</codeph>
+        configuration setting are blocked from accessing the APIs. This setting applies to all the
+        Impala-related daemons, although currently it is primarily used for HDFS to control the
+        behavior of the catalog server.
+      </p>
+    </conbody>
+
+  </concept>
+
+  <concept id="auth_to_local" rev="IMPALA-2660 CDH-40241">
+    <title>Mapping Kerberos Principals to Short Names for Impala</title>
+    <conbody>
+      <p conref="../shared/impala_common.xml#common/auth_to_local_instructions"/>
+    </conbody>
+  </concept>
+
+</concept>



[43/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_authorization.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_authorization.xml b/docs/topics/impala_authorization.xml
new file mode 100644
index 0000000..5163b2f
--- /dev/null
+++ b/docs/topics/impala_authorization.xml
@@ -0,0 +1,1621 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.1" id="authorization">
+
+  <title>Enabling Sentry Authorization for Impala</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="Sentry"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Configuring"/>
+      <data name="Category" value="Starting and Stopping"/>
+      <data name="Category" value="Users"/>
+      <data name="Category" value="Groups"/>
+      <data name="Category" value="Administrators"/>
+    </metadata>
+  </prolog>
+
+  <conbody id="sentry">
+
+    <p>
+      Authorization determines which users are allowed to access which resources, and what operations they are
+      allowed to perform. In Impala 1.1 and higher, you use the Sentry open source project for authorization.
+      Sentry adds a fine-grained authorization framework for Hadoop. By default (when authorization is not
+      enabled), Impala does all read and write operations with the privileges of the <codeph>impala</codeph> user,
+      which is suitable for a development/test environment but not for a secure production environment. When
+      authorization is enabled, Impala uses the OS user ID of the user who runs <cmdname>impala-shell</cmdname> or
+      other client program, and associates various privileges with each user.
+    </p>
+
+    <note>
+      Sentry is typically used in conjunction with Kerberos authentication, which defines which hosts are allowed
+      to connect to each server. Using the combination of Sentry and Kerberos prevents malicious users from being
+      able to connect by creating a named account on an untrusted machine. See
+      <xref href="impala_kerberos.xml#kerberos"/> for details about Kerberos authentication.
+    </note>
+
+    <p audience="PDF" outputclass="toc inpage">
+      See the following sections for details about using the Impala authorization features:
+    </p>
+  </conbody>
+
+  <concept id="sentry_priv_model">
+
+    <title>The Sentry Privilege Model</title>
+
+    <conbody>
+
+      <p>
+        Privileges can be granted on different objects in the schema. Any privilege that can be granted is
+        associated with a level in the object hierarchy. If a privilege is granted on a container object in the
+        hierarchy, the child object automatically inherits it. This is the same privilege model as Hive and other
+        database systems such as MySQL.
+      </p>
+
+      <p rev="2.3.0 collevelauth">
+        The object hierarchy for Impala covers Server, URI, Database, Table, and Column. (The Table privileges apply to views as well;
+        anywhere you specify a table name, you can specify a view name instead.)
+        Column-level authorization is available in <keyword keyref="impala23_full"/> and higher, as described in
+        <xref audience="integrated" href="sg_hive_sql.xml#concept_c2q_4qx_p4/col_level_auth_sentry"/><xref audience="standalone" href="https://www.cloudera.com/documentation/enterprise/latest/topics/sg_hive_sql.html" format="html" scope="external"/>.
+        Previously, you constructed views to query specific columns and assigned privileges
+        based on the views rather than the base tables.
+      </p>
+
+      <p>
+        A restricted set of privileges determines what you can do with each object:
+      </p>
+
+      <dl>
+        <dlentry id="select_priv">
+
+          <dt>
+            SELECT privilege
+          </dt>
+
+          <dd>
+            Lets you read data from a table or view, for example with the <codeph>SELECT</codeph> statement, the
+            <codeph>INSERT...SELECT</codeph> syntax, or <codeph>CREATE TABLE...LIKE</codeph>. Also required to
+            issue the <codeph>DESCRIBE</codeph> statement or the <codeph>EXPLAIN</codeph> statement for a query
+            against a particular table. Only objects for which a user has this privilege are shown in the output
+            for <codeph>SHOW DATABASES</codeph> and <codeph>SHOW TABLES</codeph> statements. The
+            <codeph>REFRESH</codeph> statement and <codeph>INVALIDATE METADATA</codeph> statements only access
+            metadata for tables for which the user has this privilege.
+          </dd>
+
+        </dlentry>
+
+        <dlentry id="insert_priv">
+
+          <dt>
+            INSERT privilege
+          </dt>
+
+          <dd>
+            Lets you write data to a table. Applies to the <codeph>INSERT</codeph> and <codeph>LOAD DATA</codeph>
+            statements.
+          </dd>
+
+        </dlentry>
+
+        <dlentry id="all_priv">
+
+          <dt>
+            ALL privilege
+          </dt>
+
+          <dd>
+            Lets you create or modify the object. Required to run DDL statements such as <codeph>CREATE
+            TABLE</codeph>, <codeph>ALTER TABLE</codeph>, or <codeph>DROP TABLE</codeph> for a table,
+            <codeph>CREATE DATABASE</codeph> or <codeph>DROP DATABASE</codeph> for a database, or <codeph>CREATE
+            VIEW</codeph>, <codeph>ALTER VIEW</codeph>, or <codeph>DROP VIEW</codeph> for a view. Also required for
+            the URI of the <q>location</q> parameter for the <codeph>CREATE EXTERNAL TABLE</codeph> and
+            <codeph>LOAD DATA</codeph> statements.
+<!-- Have to think about the best wording, how often to repeat, how best to conref this caveat.
+          You do not actually code the keyword <codeph>ALL</codeph> in the policy file; instead you use
+          <codeph>action=*</codeph> or shorten the right-hand portion of the rule.
+          -->
+          </dd>
+
+        </dlentry>
+      </dl>
+
+      <p>
+        Privileges can be specified for a table or view before that object actually exists. If you do not have
+        sufficient privilege to perform an operation, the error message does not disclose if the object exists or
+        not.
+      </p>
+
+      <p>
+        Originally, privileges were encoded in a policy file, stored in HDFS. This mode of operation is still an
+        option, but the emphasis of privilege management is moving towards being SQL-based. Although currently
+        Impala does not have <codeph>GRANT</codeph> or <codeph>REVOKE</codeph> statements, Impala can make use of
+        privileges assigned through <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements done through
+        Hive. The mode of operation with <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements instead of
+        the policy file requires that a special Sentry service be enabled; this service stores, retrieves, and
+        manipulates privilege information stored inside the metastore database.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="secure_startup">
+
+    <title>Starting the impalad Daemon with Sentry Authorization Enabled</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Starting and Stopping"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        To run the <cmdname>impalad</cmdname> daemon with authorization enabled, you add one or more options to the
+        <codeph>IMPALA_SERVER_ARGS</codeph> declaration in the <filepath>/etc/default/impala</filepath>
+        configuration file:
+      </p>
+
+      <ul>
+        <li>
+          The <codeph>-server_name</codeph> option turns on Sentry authorization for Impala. The authorization
+          rules refer to a symbolic server name, and you specify the name to use as the argument to the
+          <codeph>-server_name</codeph> option.
+        </li>
+
+        <li rev="1.4.0">
+          If you specify just <codeph>-server_name</codeph>, Impala uses the Sentry service for authorization,
+          relying on the results of <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements issued through
+          Hive. (This mode of operation is available in Impala 1.4.0 and higher.) Prior to Impala 1.4.0, or if you
+          want to continue storing privilege rules in the policy file, also specify the
+          <codeph>-authorization_policy_file</codeph> option as in the following item.
+        </li>
+
+        <li>
+          Specifying the <codeph>-authorization_policy_file</codeph> option in addition to
+          <codeph>-server_name</codeph> makes Impala read privilege information from a policy file, rather than
+          from the metastore database. The argument to the <codeph>-authorization_policy_file</codeph> option
+          specifies the HDFS path to the policy file that defines the privileges on different schema objects.
+        </li>
+      </ul>
+
+      <p rev="1.4.0">
+        For example, you might adapt your <filepath>/etc/default/impala</filepath> configuration to contain lines
+        like the following. To use the Sentry service rather than the policy file:
+      </p>
+
+<codeblock rev="1.4.0">IMPALA_SERVER_ARGS=" \
+-server_name=server1 \
+...
+</codeblock>
+
+      <p>
+        Or to use the policy file, as in releases prior to Impala 1.4:
+      </p>
+
+<codeblock>IMPALA_SERVER_ARGS=" \
+-authorization_policy_file=/user/hive/warehouse/auth-policy.ini \
+-server_name=server1 \
+...
+</codeblock>
+
+      <p>
+        The preceding examples set up a symbolic name of <codeph>server1</codeph> to refer to the current instance
+        of Impala. This symbolic name is used in the following ways:
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            In an environment managed by Cloudera Manager, the server name is specified through
+            <menucascade><uicontrol>Impala (Service-Wide)</uicontrol><uicontrol>Category</uicontrol><uicontrol>Advanced</uicontrol><uicontrol>Sentry Service</uicontrol></menucascade> and
+            <menucascade><uicontrol>Hive</uicontrol><uicontrol>Service-Wide</uicontrol><uicontrol>Advanced</uicontrol><uicontrol>Sentry Service</uicontrol></menucascade>.
+            The values must be the same for both, so that Impala and Hive can share the privilege rules. Restart
+            the Impala and Hive services after setting or changing this value.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            In an environment not managed by Cloudera Manager, you specify this value for the
+            <codeph>sentry.hive.server</codeph> property in the <filepath>sentry-site.xml</filepath> configuration
+            file for Hive, as well as in the <codeph>-server_name</codeph> option for <cmdname>impalad</cmdname>.
+          </p>
+          <p>
+            If the <cmdname>impalad</cmdname> daemon is not already running, start it as described in
+            <xref href="impala_processes.xml#processes"/>. If it is already running, restart it with the command
+            <codeph>sudo /etc/init.d/impala-server restart</codeph>. Run the appropriate commands on all the nodes
+            where <cmdname>impalad</cmdname> normally runs.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            If you use the mode of operation using the policy file, the rules in the <codeph>[roles]</codeph>
+            section of the policy file refer to this same <codeph>server1</codeph> name. For example, the following
+            rule sets up a role <codeph>report_generator</codeph> that lets users with that role query any table in
+            a database named <codeph>reporting_db</codeph> on a node where the <cmdname>impalad</cmdname> daemon
+            was started up with the <codeph>-server_name=server1</codeph> option:
+          </p>
+<codeblock>[roles]
+report_generator = server=server1-&gt;db=reporting_db-&gt;table=*-&gt;action=SELECT
+</codeblock>
+        </li>
+      </ul>
+
+      <p>
+        When <cmdname>impalad</cmdname> is started with one or both of the <codeph>-server_name=server1</codeph>
+        and <codeph>-authorization_policy_file</codeph> options, Impala authorization is enabled. If Impala detects
+        any errors or inconsistencies in the authorization settings or the policy file, the daemon refuses to
+        start.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="sentry_service">
+
+    <title>Using Impala with the Sentry Service (<keyword keyref="impala14"/> or higher only)</title>
+
+    <conbody>
+
+      <p>
+        When you use the Sentry service rather than the policy file, you set up privileges through
+        <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statement in either Impala or Hive, then both components
+        use those same privileges automatically. (Impala added the <codeph>GRANT</codeph> and
+        <codeph>REVOKE</codeph> statements in <keyword keyref="impala20_full"/>.)
+      </p>
+
+      <p>
+        Hive already had <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements prior to CDH 5.1, but those
+        statements were not production-ready. CDH 5.1 is the first release where those statements use the Sentry
+        framework and are considered GA level. If you used the Hive <codeph>GRANT</codeph> and
+        <codeph>REVOKE</codeph> statements prior to CDH 5.1, you must set up these privileges with the CDH 5.1
+        versions of <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> to take advantage of Sentry authorization.
+      </p>
+
+      <p>
+        For information about using the updated Hive <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements,
+        see
+<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Security-Guide/cdh_sg_sentry_service.html -->
+        <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_sentry_service.html" scope="external" format="html">Sentry
+        service</xref> topic in the <cite>CDH 5 Security Guide</cite>.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="security_policy_file">
+
+    <title>Using Impala with the Sentry Policy File</title>
+
+    <conbody>
+
+      <p>
+        The policy file is a file that you put in a designated location in HDFS, and is read during the startup of
+        the <cmdname>impalad</cmdname> daemon when you specify both the <codeph>-server_name</codeph> and
+        <codeph>-authorization_policy_file</codeph> startup options. It controls which objects (databases, tables,
+        and HDFS directory paths) can be accessed by the user who connects to <cmdname>impalad</cmdname>, and what
+        operations that user can perform on the objects.
+      </p>
+
+      <note rev="1.4.0">
+        <p rev="1.4.0">
+          In <ph rev="upstream">CDH 5</ph> and higher, <ph rev="upstream">Cloudera</ph> recommends
+          managing privileges through SQL statements, as described in
+          <xref href="impala_authorization.xml#sentry_service"/>. If you are still using policy files, plan to
+          migrate to the new approach some time in the future.
+        </p>
+      </note>
+
+      <p>
+        The location of the policy file is listed in the <filepath>auth-site.xml</filepath> configuration file. To
+        minimize overhead, the security information from this file is cached by each <cmdname>impalad</cmdname>
+        daemon and refreshed automatically, with a default interval of 5 minutes. After making a substantial change
+        to security policies, restart all Impala daemons to pick up the changes immediately.
+      </p>
+
+      <p outputclass="toc inpage"/>
+    </conbody>
+
+    <concept id="security_policy_file_details">
+
+      <title>Policy File Location and Format</title>
+
+      <conbody>
+
+        <p>
+          The policy file uses the familiar <codeph>.ini</codeph> format, divided into the major sections
+          <codeph>[groups]</codeph> and <codeph>[roles]</codeph>. There is also an optional
+          <codeph>[databases]</codeph> section, which allows you to specify a specific policy file for a particular
+          database, as explained in <xref href="#security_multiple_policy_files"/>. Another optional section,
+          <codeph>[users]</codeph>, allows you to override the OS-level mapping of users to groups; that is an
+          advanced technique primarily for testing and debugging, and is beyond the scope of this document.
+        </p>
+
+        <p>
+          In the <codeph>[groups]</codeph> section, you define various categories of users and select which roles
+          are associated with each category. The group and usernames correspond to Linux groups and users on the
+          server where the <cmdname>impalad</cmdname> daemon runs.
+        </p>
+
+        <p>
+          The group and usernames in the <codeph>[groups]</codeph> section correspond to Linux groups and users on
+          the server where the <cmdname>impalad</cmdname> daemon runs. When you access Impala through the
+          <cmdname>impalad</cmdname> interpreter, for purposes of authorization, the user is the logged-in Linux
+          user and the groups are the Linux groups that user is a member of. When you access Impala through the
+          ODBC or JDBC interfaces, the user and password specified through the connection string are used as login
+          credentials for the Linux server, and authorization is based on that username and the associated Linux
+          group membership.
+        </p>
+
+        <p>
+          In the <codeph>[roles]</codeph> section, you a set of roles. For each role, you specify precisely the set
+          of privileges is available. That is, which objects users with that role can access, and what operations
+          they can perform on those objects. This is the lowest-level category of security information; the other
+          sections in the policy file map the privileges to higher-level divisions of groups and users. In the
+          <codeph>[groups]</codeph> section, you specify which roles are associated with which groups. The group
+          and usernames correspond to Linux groups and users on the server where the <cmdname>impalad</cmdname>
+          daemon runs. The privileges are specified using patterns like:
+<codeblock>server=<varname>server_name</varname>-&gt;db=<varname>database_name</varname>-&gt;table=<varname>table_name</varname>-&gt;action=SELECT
+server=<varname>server_name</varname>-&gt;db=<varname>database_name</varname>-&gt;table=<varname>table_name</varname>-&gt;action=CREATE
+server=<varname>server_name</varname>-&gt;db=<varname>database_name</varname>-&gt;table=<varname>table_name</varname>-&gt;action=ALL
+</codeblock>
+          For the <varname>server_name</varname> value, substitute the same symbolic name you specify with the
+          <cmdname>impalad</cmdname> <codeph>-server_name</codeph> option. You can use <codeph>*</codeph> wildcard
+          characters at each level of the privilege specification to allow access to all such objects. For example:
+<codeblock>server=impala-host.example.com-&gt;db=default-&gt;table=t1-&gt;action=SELECT
+server=impala-host.example.com-&gt;db=*-&gt;table=*-&gt;action=CREATE
+server=impala-host.example.com-&gt;db=*-&gt;table=audit_log-&gt;action=SELECT
+server=impala-host.example.com-&gt;db=default-&gt;table=t1-&gt;action=*
+</codeblock>
+        </p>
+
+        <p>
+          When authorization is enabled, Impala uses the policy file as a <i>whitelist</i>, representing every
+          privilege available to any user on any object. That is, only operations specified for the appropriate
+          combination of object, role, group, and user are allowed; all other operations are not allowed. If a
+          group or role is defined multiple times in the policy file, the last definition takes precedence.
+        </p>
+
+        <p>
+          To understand the notion of whitelisting, set up a minimal policy file that does not provide any
+          privileges for any object. When you connect to an Impala node where this policy file is in effect, you
+          get no results for <codeph>SHOW DATABASES</codeph>, and an error when you issue any <codeph>SHOW
+          TABLES</codeph>, <codeph>USE <varname>database_name</varname></codeph>, <codeph>DESCRIBE
+          <varname>table_name</varname></codeph>, <codeph>SELECT</codeph>, and or other statements that expect to
+          access databases or tables, even if the corresponding databases and tables exist.
+        </p>
+
+        <p>
+          The contents of the policy file are cached, to avoid a performance penalty for each query. The policy
+          file is re-checked by each <cmdname>impalad</cmdname> node every 5 minutes. When you make a
+          non-time-sensitive change such as adding new privileges or new users, you can let the change take effect
+          automatically a few minutes later. If you remove or reduce privileges, and want the change to take effect
+          immediately, restart the <cmdname>impalad</cmdname> daemon on all nodes, again specifying the
+          <codeph>-server_name</codeph> and <codeph>-authorization_policy_file</codeph> options so that the rules
+          from the updated policy file are applied.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept id="security_examples">
+
+      <title>Examples of Policy File Rules for Security Scenarios</title>
+
+      <conbody>
+
+        <p>
+          The following examples show rules that might go in the policy file to deal with various
+          authorization-related scenarios. For illustration purposes, this section shows several very small policy
+          files with only a few rules each. In your environment, typically you would define many roles to cover all
+          the scenarios involving your own databases, tables, and applications, and a smaller number of groups,
+          whose members are given the privileges from one or more roles.
+        </p>
+
+        <example id="sec_ex_unprivileged">
+
+          <title>A User with No Privileges</title>
+
+          <p>
+            If a user has no privileges at all, that user cannot access any schema objects in the system. The error
+            messages do not disclose the names or existence of objects that the user is not authorized to read.
+          </p>
+
+          <p>
+<!--        This example demonstrates the lack of privileges using a blank policy file, so no users have any privileges. -->
+            This is the experience you want a user to have if they somehow log into a system where they are not an
+            authorized Impala user. In a real deployment with a filled-in policy file, a user might have no
+            privileges because they are not a member of any of the relevant groups mentioned in the policy file.
+          </p>
+
+<!-- Have the raw material but not formatted into easily digestible example. Do for first 1.1 doc refresh.
+<codeblock></codeblock> -->
+
+        </example>
+
+        <example id="sec_ex_superuser">
+
+          <title>Examples of Privileges for Administrative Users</title>
+
+          <p>
+            When an administrative user has broad access to tables or databases, the associated rules in the
+            <codeph>[roles]</codeph> section typically use wildcards and/or inheritance. For example, in the
+            following sample policy file, <codeph>db=*</codeph> refers to all databases and
+            <codeph>db=*-&gt;table=*</codeph> refers to all tables in all databases.
+          </p>
+
+          <p>
+            Omitting the rightmost portion of a rule means that the privileges apply to all the objects that could
+            be specified there. For example, in the following sample policy file, the
+            <codeph>all_databases</codeph> role has all privileges for all tables in all databases, while the
+            <codeph>one_database</codeph> role has all privileges for all tables in one specific database. The
+            <codeph>all_databases</codeph> role does not grant privileges on URIs, so a group with that role could
+            not issue a <codeph>CREATE TABLE</codeph> statement with a <codeph>LOCATION</codeph> clause. The
+            <codeph>entire_server</codeph> role has all privileges on both databases and URIs within the server.
+          </p>
+
+<codeblock>[groups]
+supergroup = all_databases
+
+[roles]
+read_all_tables = server=server1-&gt;db=*-&gt;table=*-&gt;action=SELECT
+all_tables = server=server1-&gt;db=*-&gt;table=*
+all_databases = server=server1-&gt;db=*
+one_database = server=server1-&gt;db=test_db
+entire_server = server=server1
+</codeblock>
+
+        </example>
+
+        <example id="sec_ex_detailed">
+
+          <title>A User with Privileges for Specific Databases and Tables</title>
+
+          <p>
+            If a user has privileges for specific tables in specific databases, the user can access those things
+            but nothing else. They can see the tables and their parent databases in the output of <codeph>SHOW
+            TABLES</codeph> and <codeph>SHOW DATABASES</codeph>, <codeph>USE</codeph> the appropriate databases,
+            and perform the relevant actions (<codeph>SELECT</codeph> and/or <codeph>INSERT</codeph>) based on the
+            table privileges. To actually create a table requires the <codeph>ALL</codeph> privilege at the
+            database level, so you might define separate roles for the user that sets up a schema and other users
+            or applications that perform day-to-day operations on the tables.
+          </p>
+
+          <p>
+            The following sample policy file shows some of the syntax that is appropriate as the policy file grows,
+            such as the <codeph>#</codeph> comment syntax, <codeph>\</codeph> continuation syntax, and comma
+            separation for roles assigned to groups or privileges assigned to roles.
+          </p>
+
+<codeblock>[groups]
+cloudera = training_sysadmin, instructor
+visitor = student
+
+[roles]
+training_sysadmin = server=server1-&gt;db=training, \
+server=server1-&gt;db=instructor_private, \
+server=server1-&gt;db=lesson_development
+instructor = server=server1-&gt;db=training-&gt;table=*-&gt;action=*, \
+server=server1-&gt;db=instructor_private-&gt;table=*-&gt;action=*, \
+server=server1-&gt;db=lesson_development-&gt;table=lesson*
+# This particular course is all about queries, so the students can SELECT but not INSERT or CREATE/DROP.
+student = server=server1-&gt;db=training-&gt;table=lesson_*-&gt;action=SELECT
+</codeblock>
+
+        </example>
+
+<!--
+<example id="sec_ex_superuser_single_db">
+<title>A User with Full Privileges for a Specific Database</title>
+<p>
+
+</p>
+<codeblock></codeblock>
+</example>
+
+<example id="sec_ex_readonly_single_db">
+<title>A User with Read-Only Privileges for a Specific Database</title>
+<p>
+
+</p>
+<codeblock></codeblock>
+    <p>
+      If a user has <codeph>SELECT</codeph> privilege for a database, they can issue a <codeph>USE</codeph> statement
+      for that database. Whether or not they can access tables within the database depends on further privileges
+      defined at the table level.
+    </p>
+
+<codeblock></codeblock>
+
+</example>
+
+<example id="sec_ex_superuser_single_table">
+<title>A User with Full Privileges for a Specific Table</title>
+    <p>
+      If a user has <codeph>SELECT</codeph> privilege for a table, they can query, describe, or explain queries for
+      that table.
+    </p>
+
+<codeblock></codeblock>
+</example>
+
+<example id="sec_ex_load_data">
+<title>A User with Privileges to Load Data but not Read Data</title>
+
+    <p>
+      If a user has <codeph>INSERT</codeph> privilege for a table, they can write to the table if it already exists.
+      They cannot create or alter the table; those operations require the <codeph>ALL</codeph> privilege.
+    </p>
+<codeblock></codeblock>
+</example>
+-->
+
+        <example id="sec_ex_external_files">
+
+          <title>Privileges for Working with External Data Files</title>
+
+          <p>
+            When data is being inserted through the <codeph>LOAD DATA</codeph> statement, or is referenced from an
+            HDFS location outside the normal Impala database directories, the user also needs appropriate
+            permissions on the URIs corresponding to those HDFS locations.
+          </p>
+
+          <p>
+            In this sample policy file:
+          </p>
+
+          <ul>
+            <li>
+              The <codeph>external_table</codeph> role lets us insert into and query the Impala table,
+              <codeph>external_table.sample</codeph>.
+            </li>
+
+            <li>
+              The <codeph>staging_dir</codeph> role lets us specify the HDFS path
+              <filepath>/user/cloudera/external_data</filepath> with the <codeph>LOAD DATA</codeph> statement.
+              Remember, when Impala queries or loads data files, it operates on all the files in that directory,
+              not just a single file, so any Impala <codeph>LOCATION</codeph> parameters refer to a directory
+              rather than an individual file.
+            </li>
+
+            <li>
+              We included the IP address and port of the Hadoop name node in the HDFS URI of the
+              <codeph>staging_dir</codeph> rule. We found those details in
+              <filepath>/etc/hadoop/conf/core-site.xml</filepath>, under the <codeph>fs.default.name</codeph>
+              element. That is what we use in any roles that specify URIs (that is, the locations of directories in
+              HDFS).
+            </li>
+
+            <li>
+              We start this example after the table <codeph>external_table.sample</codeph> is already created. In
+              the policy file for the example, we have already taken away the <codeph>external_table_admin</codeph>
+              role from the <codeph>cloudera</codeph> group, and replaced it with the lesser-privileged
+              <codeph>external_table</codeph> role.
+            </li>
+
+            <li>
+              We assign privileges to a subdirectory underneath <filepath>/user/cloudera</filepath> in HDFS,
+              because such privileges also apply to any subdirectories underneath. If we had assigned privileges to
+              the parent directory <filepath>/user/cloudera</filepath>, it would be too likely to mess up other
+              files by specifying a wrong location by mistake.
+            </li>
+
+            <li>
+              The <codeph>cloudera</codeph> under the <codeph>[groups]</codeph> section refers to the
+              <codeph>cloudera</codeph> group. (In the demo VM used for this example, there is a
+              <codeph>cloudera</codeph> user that is a member of a <codeph>cloudera</codeph> group.)
+            </li>
+          </ul>
+
+          <p>
+            Policy file:
+          </p>
+
+<codeblock>[groups]
+cloudera = external_table, staging_dir
+
+[roles]
+external_table_admin = server=server1-&gt;db=external_table
+external_table = server=server1-&gt;db=external_table-&gt;table=sample-&gt;action=*
+staging_dir = server=server1-&gt;uri=hdfs://127.0.0.1:8020/user/cloudera/external_data-&gt;action=*
+</codeblock>
+
+          <p>
+            <cmdname>impala-shell</cmdname> session:
+          </p>
+
+<codeblock>[localhost:21000] &gt; use external_table;
+Query: use external_table
+[localhost:21000] &gt; show tables;
+Query: show tables
+Query finished, fetching results ...
++--------+
+| name   |
++--------+
+| sample |
++--------+
+Returned 1 row(s) in 0.02s
+
+[localhost:21000] &gt; select * from sample;
+Query: select * from sample
+Query finished, fetching results ...
++-----+
+| x   |
++-----+
+| 1   |
+| 5   |
+| 150 |
++-----+
+Returned 3 row(s) in 1.04s
+
+[localhost:21000] &gt; load data inpath '/user/cloudera/external_data' into table sample;
+Query: load data inpath '/user/cloudera/external_data' into table sample
+Query finished, fetching results ...
++----------------------------------------------------------+
+| summary                                                  |
++----------------------------------------------------------+
+| Loaded 1 file(s). Total files in destination location: 2 |
++----------------------------------------------------------+
+Returned 1 row(s) in 0.26s
+[localhost:21000] &gt; select * from sample;
+Query: select * from sample
+Query finished, fetching results ...
++-------+
+| x     |
++-------+
+| 2     |
+| 4     |
+| 6     |
+| 8     |
+| 64738 |
+| 49152 |
+| 1     |
+| 5     |
+| 150   |
++-------+
+Returned 9 row(s) in 0.22s
+
+[localhost:21000] &gt; load data inpath '/user/cloudera/unauthorized_data' into table sample;
+Query: load data inpath '/user/cloudera/unauthorized_data' into table sample
+ERROR: AuthorizationException: User 'cloudera' does not have privileges to access: hdfs://127.0.0.1:8020/user/cloudera/unauthorized_data
+</codeblock>
+
+        </example>
+
+        <example audience="Cloudera" id="sec_ex_views" rev="2.3.0 collevelauth">
+
+          <title>Controlling Access at the Column Level through Views</title>
+
+          <p>
+            If a user has <codeph>SELECT</codeph> privilege for a view, they can query the view, even if they do
+            not have any privileges on the underlying table. To see the details about the underlying table through
+            <codeph>EXPLAIN</codeph> or <codeph>DESCRIBE FORMATTED</codeph> statements on the view, the user must
+            also have <codeph>SELECT</codeph> privilege for the underlying table.
+          </p>
+
+          <note type="important">
+            <p>
+              The types of data that are considered sensitive and confidential differ depending on the jurisdiction
+              the type of industry, or both. For fine-grained access controls, set up appropriate privileges based
+              on all applicable laws and regulations.
+            </p>
+            <p>
+              Be careful using the <codeph>ALTER VIEW</codeph> statement to point an existing view at a different
+              base table or a new set of columns that includes sensitive or restricted data. Make sure that any
+              users who have <codeph>SELECT</codeph> privilege on the view do not gain access to any additional
+              information they are not authorized to see.
+            </p>
+          </note>
+
+          <p>
+            The following example shows how a system administrator could set up a table containing some columns
+            with sensitive information, then create a view that only exposes the non-confidential columns.
+          </p>
+
+<codeblock>[localhost:21000] &gt; create table sensitive_info
+                &gt; (
+                &gt;   name string,
+                &gt;   address string,
+                &gt;   credit_card string,
+                &gt;   taxpayer_id string
+                &gt; );
+[localhost:21000] &gt; create view name_address_view as select name, address from sensitive_info;
+</codeblock>
+
+          <p>
+            Then the following policy file specifies read-only privilege for that view, without authorizing access
+            to the underlying table:
+          </p>
+
+<codeblock>[groups]
+cloudera = view_only_privs
+
+[roles]
+view_only_privs = server=server1-&gt;db=reports-&gt;table=name_address_view-&gt;action=SELECT
+</codeblock>
+
+          <p>
+            Thus, a user with the <codeph>view_only_privs</codeph> role could access through Impala queries the
+            basic information but not the sensitive information, even if both kinds of information were part of the
+            same data file.
+          </p>
+
+          <p>
+            You might define other views to allow users from different groups to query different sets of columns.
+          </p>
+
+        </example>
+
+        <example id="sec_sysadmin">
+
+          <title>Separating Administrator Responsibility from Read and Write Privileges</title>
+
+          <p>
+            Remember that to create a database requires full privilege on that database, while day-to-day
+            operations on tables within that database can be performed with lower levels of privilege on specific
+            table. Thus, you might set up separate roles for each database or application: an administrative one
+            that could create or drop the database, and a user-level one that can access only the relevant tables.
+          </p>
+
+          <p>
+            For example, this policy file divides responsibilities between users in 3 different groups:
+          </p>
+
+          <ul>
+            <li>
+              Members of the <codeph>supergroup</codeph> group have the <codeph>training_sysadmin</codeph> role and
+              so can set up a database named <codeph>training</codeph>.
+            </li>
+
+            <li>
+              Members of the <codeph>cloudera</codeph> group have the <codeph>instructor</codeph> role and so can
+              create, insert into, and query any tables in the <codeph>training</codeph> database, but cannot
+              create or drop the database itself.
+            </li>
+
+            <li>
+              Members of the <codeph>visitor</codeph> group have the <codeph>student</codeph> role and so can query
+              those tables in the <codeph>training</codeph> database.
+            </li>
+          </ul>
+
+<codeblock>[groups]
+supergroup = training_sysadmin
+cloudera = instructor
+visitor = student
+
+[roles]
+training_sysadmin = server=server1-&gt;db=training
+instructor = server=server1-&gt;db=training-&gt;table=*-&gt;action=*
+student = server=server1-&gt;db=training-&gt;table=*-&gt;action=SELECT
+</codeblock>
+
+        </example>
+      </conbody>
+    </concept>
+
+    <concept id="security_multiple_policy_files">
+
+      <title>Using Multiple Policy Files for Different Databases</title>
+
+      <conbody>
+
+        <p>
+          For an Impala cluster with many databases being accessed by many users and applications, it might be
+          cumbersome to update the security policy file for each privilege change or each new database, table, or
+          view. You can allow security to be managed separately for individual databases, by setting up a separate
+          policy file for each database:
+        </p>
+
+        <ul>
+          <li>
+            Add the optional <codeph>[databases]</codeph> section to the main policy file.
+          </li>
+
+          <li>
+            Add entries in the <codeph>[databases]</codeph> section for each database that has its own policy file.
+          </li>
+
+          <li>
+            For each listed database, specify the HDFS path of the appropriate policy file.
+          </li>
+        </ul>
+
+        <p>
+          For example:
+        </p>
+
+<codeblock>[databases]
+# Defines the location of the per-DB policy files for the 'customers' and 'sales' databases.
+customers = hdfs://ha-nn-uri/etc/access/customers.ini
+sales = hdfs://ha-nn-uri/etc/access/sales.ini
+</codeblock>
+
+        <p>
+          To enable URIs in per-DB policy files, add the following string in the Cloudera Manager field
+          <uicontrol>Impala Service Environment Advanced Configuration Snippet (Safety Valve)</uicontrol>:
+        </p>
+
+<codeblock>JAVA_TOOL_OPTIONS="-Dsentry.allow.uri.db.policyfile=true"
+</codeblock>
+
+        <note type="important">
+          Enabling URIs in per-DB policy files introduces a security risk by allowing the owner of the db-level
+          policy file to grant himself/herself load privileges to anything the <codeph>impala</codeph> user has
+          read permissions for in HDFS (including data in other databases controlled by different db-level policy
+          files).
+        </note>
+      </conbody>
+    </concept>
+  </concept>
+
+  <concept id="security_schema">
+
+    <title>Setting Up Schema Objects for a Secure Impala Deployment</title>
+
+    <conbody>
+
+      <p>
+        Remember that in your role definitions, you specify privileges at the level of individual databases and
+        tables, or all databases or all tables within a database. To simplify the structure of these rules, plan
+        ahead of time how to name your schema objects so that data with different authorization requirements is
+        divided into separate databases.
+      </p>
+
+      <p>
+        If you are adding security on top of an existing Impala deployment, remember that you can rename tables or
+        even move them between databases using the <codeph>ALTER TABLE</codeph> statement. In Impala, creating new
+        databases is a relatively inexpensive operation, basically just creating a new directory in HDFS.
+      </p>
+
+      <p>
+        You can also plan the security scheme and set up the policy file before the actual schema objects named in
+        the policy file exist. Because the authorization capability is based on whitelisting, a user can only
+        create a new database or table if the required privilege is already in the policy file: either by listing
+        the exact name of the object being created, or a <codeph>*</codeph> wildcard to match all the applicable
+        objects within the appropriate container.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="security_privileges">
+
+    <title>Privilege Model and Object Hierarchy</title>
+
+    <conbody>
+
+      <p>
+        Privileges can be granted on different objects in the schema. Any privilege that can be granted is
+        associated with a level in the object hierarchy. If a privilege is granted on a container object in the
+        hierarchy, the child object automatically inherits it. This is the same privilege model as Hive and other
+        database systems such as MySQL.
+      </p>
+
+      <p>
+        The kinds of objects in the schema hierarchy are:
+      </p>
+
+<codeblock>Server
+URI
+Database
+  Table
+</codeblock>
+
+      <p>
+        The server name is specified by the <codeph>-server_name</codeph> option when <cmdname>impalad</cmdname>
+        starts. Specify the same name for all <cmdname>impalad</cmdname> nodes in the cluster.
+      </p>
+
+      <p>
+        URIs represent the HDFS paths you specify as part of statements such as <codeph>CREATE EXTERNAL
+        TABLE</codeph> and <codeph>LOAD DATA</codeph>. Typically, you specify what look like UNIX paths, but these
+        locations can also be prefixed with <codeph>hdfs://</codeph> to make clear that they are really URIs. To
+        set privileges for a URI, specify the name of a directory, and the privilege applies to all the files in
+        that directory and any directories underneath it.
+      </p>
+
+      <p rev="2.3.0 collevelauth">
+        In <keyword keyref="impala23_full"/> and higher, you can specify privileges for individual columns,
+        as described in
+        <xref audience="integrated" href="sg_hive_sql.xml#concept_c2q_4qx_p4/col_level_auth_sentry"/><xref audience="standalone" href="https://www.cloudera.com/documentation/enterprise/latest/topics/sg_hive_sql.html" format="html" scope="external"/>.
+        Formerly, to specify
+        read privileges at this level, you created a view that queried specific columns and/or partitions from a base
+        table, and gave <codeph>SELECT</codeph> privilege on the view but not the underlying table.
+      </p>
+
+      <p>
+        URIs must start with either <codeph>hdfs://</codeph> or <codeph>file://</codeph>. If a URI starts with
+        anything else, it will cause an exception and the policy file will be invalid. When defining URIs for HDFS,
+        you must also specify the NameNode. For example:
+<codeblock>data_read = server=server1-&gt;uri=file:///path/to/dir, \
+server=server1-&gt;uri=hdfs://namenode:port/path/to/dir
+</codeblock>
+        <note type="warning">
+          <p>
+            Because the NameNode host and port must be specified, Cloudera strongly recommends you use High
+            Availability (HA). This ensures that the URI will remain constant even if the namenode changes.
+          </p>
+<codeblock>data_read = server=server1-&gt;uri=file:///path/to/dir,\ server=server1-&gt;uri=hdfs://ha-nn-uri/path/to/dir
+</codeblock>
+        </note>
+      </p>
+
+<!-- Experiment with replacing my original copied table with a conref'ed version of Ambreen's from the Security Guide. -->
+
+<!--
+          <table>
+          <title>Sentry privilege types and objects they apply to</title>
+          <tgroup cols="2">
+              <colspec colnum="1" colname="col1"/>
+              <colspec colnum="2" colname="col2"/>
+              <tbody>
+                  <row>
+                      <entry>Privilege</entry>
+                      <entry>Object</entry>
+                  </row>
+                  <row>
+                      <entry>INSERT</entry>
+                      <entry>TABLE, URI</entry>
+                  </row>
+                  <row>
+                      <entry>SELECT</entry>
+                      <entry>TABLE, VIEW, URI</entry>
+                  </row>
+                  <row>
+                      <entry>ALL</entry>
+                      <entry>SERVER, DB, URI</entry>
+                  </row>
+              </tbody>
+          </tgroup>
+          </table>
+-->
+
+      <table conref="../shared/impala_common.xml#common/sentry_privileges_objects">
+        <tgroup cols="2">
+          <colspec colnum="1" colname="col1" colwidth="1*"/>
+          <tbody>
+            <row>
+              <entry/>
+            </row>
+          </tbody>
+        </tgroup>
+      </table>
+
+      <note>
+        <p>
+          Although this document refers to the <codeph>ALL</codeph> privilege, currently if you use the policy file
+          mode, you do not use the actual keyword <codeph>ALL</codeph> in the policy file. When you code role
+          entries in the policy file:
+        </p>
+        <ul>
+          <li>
+            To specify the <codeph>ALL</codeph> privilege for a server, use a role like
+            <codeph>server=<varname>server_name</varname></codeph>.
+          </li>
+
+          <li>
+            To specify the <codeph>ALL</codeph> privilege for a database, use a role like
+            <codeph>server=<varname>server_name</varname>-&gt;db=<varname>database_name</varname></codeph>.
+          </li>
+
+          <li>
+            To specify the <codeph>ALL</codeph> privilege for a table, use a role like
+            <codeph>server=<varname>server_name</varname>-&gt;db=<varname>database_name</varname>-&gt;table=<varname>table_name</varname>-&gt;action=*</codeph>.
+          </li>
+        </ul>
+      </note>
+
+      <draft-comment author="ambreen.kazi" translate="no">The Privilege Model lives at:
+https://wiki.cloudera.com/pages/viewpage.action?pageId=24919851</draft-comment>
+
+      <table>
+        <tgroup cols="4">
+          <colspec colnum="1" colname="col1" colwidth="1.31*"/>
+          <colspec colnum="2" colname="col2" colwidth="1.17*"/>
+          <colspec colnum="3" colname="col3" colwidth="1*"/>
+          <colspec colname="newCol4" colnum="4" colwidth="1*"/>
+          <thead>
+            <row>
+              <entry>
+                Operation
+              </entry>
+              <entry>
+                Scope
+              </entry>
+              <entry>
+                Privileges
+              </entry>
+              <entry>
+                URI
+              </entry>
+            </row>
+          </thead>
+          <tbody>
+            <row conref="../shared/impala_common.xml#common/explain_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/load_data_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/create_database_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/drop_database_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/create_table_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/drop_table_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/describe_table_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/alter_table_add_columns_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/alter_table_replace_columns_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/alter_table_change_column_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/alter_table_rename_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/alter_table_set_tblproperties_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/alter_table_set_fileformat_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/alter_table_set_location_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/alter_table_add_partition_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/alter_table_add_partition_location_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/alter_table_drop_partition_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/alter_table_partition_set_fileformat_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/alter_table_set_serdeproperties_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/create_view_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/drop_view_privs">
+              <entry/>
+            </row>
+            <row id="alter_view_privs">
+              <entry>
+                ALTER VIEW
+              </entry>
+              <entry rev="2.3.0 collevelauth">
+                You need <codeph>ALL</codeph> privilege on the named view <ph rev="1.4.0">and the parent
+                database</ph>, plus <codeph>SELECT</codeph> privilege for any tables or views referenced by the
+                view query. Once the view is created or altered by a high-privileged system administrator, it can
+                be queried by a lower-privileged user who does not have full query privileges for the base tables.
+              </entry>
+              <entry>
+                ALL, SELECT
+              </entry>
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/alter_table_set_location_privs">
+              <entry/>
+            </row>
+            <row id="create_external_table_privs">
+              <entry>
+                CREATE EXTERNAL TABLE
+              </entry>
+              <entry>
+                Database (ALL), URI (SELECT)
+              </entry>
+              <entry>
+                ALL, SELECT
+              </entry>
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/select_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/use_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/create_function_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/drop_function_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/refresh_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/invalidate_metadata_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/invalidate_metadata_table_privs">
+              <entry/>
+            </row>
+            <row conref="../shared/impala_common.xml#common/compute_stats_privs">
+              <entry/>
+            </row>
+            <row id="show_table_stats_privs">
+              <entry>
+                SHOW TABLE STATS, SHOW PARTITIONS
+              </entry>
+              <entry>
+                TABLE
+              </entry>
+              <entry>
+                SELECT/INSERT
+              </entry>
+              <entry/>
+            </row>
+            <row>
+              <entry id="show_column_stats_privs">
+                SHOW COLUMN STATS
+              </entry>
+              <entry>
+                TABLE
+              </entry>
+              <entry>
+                SELECT/INSERT
+              </entry>
+              <entry/>
+            </row>
+            <row>
+              <entry id="show_functions_privs">
+                SHOW FUNCTIONS
+              </entry>
+              <entry>
+                DATABASE
+              </entry>
+              <entry>
+                SELECT
+              </entry>
+              <entry/>
+            </row>
+            <row id="show_tables_privs">
+              <entry>
+                SHOW TABLES
+              </entry>
+              <entry/>
+              <entry>
+                No special privileges needed to issue the statement, but only shows objects you are authorized for
+              </entry>
+              <entry/>
+            </row>
+            <row id="show_databases_privs">
+              <entry>
+                SHOW DATABASES, SHOW SCHEMAS
+              </entry>
+              <entry/>
+              <entry>
+                No special privileges needed to issue the statement, but only shows objects you are authorized for
+              </entry>
+              <entry/>
+            </row>
+          </tbody>
+        </tgroup>
+      </table>
+
+<!-- Original table, now replaced by content conref'ed from Security Guide
+<table>
+<title>Sentry privileges for Impala SQL operations</title>
+          <tgroup cols="3">
+              <colspec colnum="1" colname="col1" colwidth="*"/>
+              <colspec colnum="2" colname="col2" colwidth="*"/>
+              <colspec colnum="3" colname="col3" colwidth="4*"/>
+<thead>
+<row>
+<entry>Impala SQL Operation</entry>
+<entry>Privileges Required</entry>
+<entry>Object on which Privileges Required</entry>
+</row>
+</thead>
+<tbody>
+<row>
+<entry>EXPLAIN</entry>
+<entry>SELECT</entry>
+<entry>Table</entry>
+</row>
+<row>
+<entry>LOAD DATA</entry>
+<entry>INSERT, SELECT</entry>
+<entry>Table (INSERT) and URI (ALL); write privilege is required on the URI because the data files are physically moved from there</entry>
+</row>
+<row>
+<entry>CREATE DATABASE</entry>
+<entry>ALL</entry>
+<entry><ph rev="1.4.0">Server</ph></entry>
+</row>
+<row>
+<entry>DROP DATABASE</entry>
+<entry>ALL</entry>
+<entry><ph rev="1.4.0">Server</ph></entry>
+</row>
+<row>
+<entry>CREATE TABLE</entry>
+<entry>ALL</entry>
+<entry><ph rev="1.4.0">Database</ph></entry>
+</row>
+<row>
+<entry>DROP TABLE</entry>
+<entry>ALL</entry>
+<entry><ph rev="1.4.0">Database</ph></entry>
+</row>
+<row>
+<entry>DESCRIBE TABLE</entry>
+<entry>SELECT or INSERT</entry>
+<entry>Table</entry>
+</row>
+<row>
+<entry>ALTER TABLE</entry>
+<entry>ALL</entry>
+<entry><ph rev="1.4.0">Database</ph></entry>
+</row>
+<row>
+<entry>SHOW DATABASES</entry>
+<entry>Any privilege</entry>
+<entry>Any object in the database; only databases for which the user has privileges on some object are shown</entry>
+</row>
+<row>
+<entry>SHOW TABLES</entry>
+<entry>Any privilege</entry>
+<entry>Any table in the database; only tables for which the user has privileges are shown</entry>
+</row>
+<row>
+<entry>CREATE VIEW</entry>
+<entry>ALL, SELECT</entry>
+<entry rev="2.3.0 collevelauth">
+You need <codeph>ALL</codeph> privilege on the named view <ph rev="1.4.0">and the parent database</ph>, plus <codeph>SELECT</codeph> privilege for any tables or views referenced by the view query.
+Once the view is created or altered by a high-privileged system administrator,
+it can be queried by a lower-privileged user who does not have full query privileges for the base tables.
+</entry>
+</row>
+<row>
+<entry>DROP VIEW</entry>
+<entry>ALL</entry>
+<entry><ph rev="1.4.0">Database</ph></entry>
+</row>
+<row>
+<entry>ALTER VIEW</entry>
+<entry>ALL, SELECT</entry>
+<entry rev="2.3.0 collevelauth">
+You need <codeph>ALL</codeph> privilege on the named view <ph rev="1.4.0">and the parent database</ph>, plus <codeph>SELECT</codeph> privilege for any tables or views referenced by the view query.
+Once the view is created or altered by a high-privileged system administrator,
+it can be queried by a lower-privileged user who does not have full query privileges for the base tables.
+</entry>
+</row>
+<row>
+<entry>ALTER TABLE LOCATION</entry>
+<entry>ALL</entry>
+<entry>Table, URI</entry>
+</row>
+<row>
+<entry>CREATE EXTERNAL TABLE</entry>
+<entry>ALL, SELECT</entry>
+<entry>Database (ALL), URI (SELECT)</entry>
+</row>
+<row>
+<entry>SELECT</entry>
+<entry>SELECT</entry>
+<entry rev="2.3.0 collevelauth">Table, View; you can have SELECT privilege for a view without having SELECT privilege
+for the underlying tables. See
+<xref audience="integrated" href="sg_hive_sql.xml#concept_c2q_4qx_p4/col_level_auth_sentry"/><xref audience="standalone" href="https://www.cloudera.com/documentation/enterprise/latest/topics/sg_hive_sql.html" format="html" scope="external"/>
+for details about column-level authorization.
+Before column-level authorization was added in <keyword keyref="impala23_full"/> the way to implement
+column-level security was by creating views that referenced particular sets of columns.</entry>
+</row>
+<row>
+<entry>USE</entry>
+<entry>Any privilege</entry>
+<entry>Any object in the database</entry>
+</row>
+<row rev="1.2">
+<entry>CREATE FUNCTION</entry>
+<entry>ALL</entry>
+<entry>Server</entry>
+</row>
+<row rev="1.2">
+<entry>DROP FUNCTION</entry>
+<entry>ALL</entry>
+<entry>Server</entry>
+</row>
+<row>
+<entry>REFRESH</entry>
+<entry>ALL</entry>
+<entry>Table</entry>
+</row>
+<row>
+<entry>INVALIDATE METADATA</entry>
+<entry>ALL</entry>
+<entry>Server</entry>
+</row>
+<row rev="1.2.2">
+<entry>COMPUTE STATS</entry>
+<entry>ALL</entry>
+<entry>Table</entry>
+</row>
+<row>
+<entry>SHOW TABLE STATS, SHOW PARTITIONS</entry>
+<entry>?</entry>
+<entry>?</entry>
+</row>
+<row>
+<entry>SHOW COLUMN STATS</entry>
+<entry>?</entry>
+<entry>?</entry>
+</row>
+<row>
+<entry>SHOW FUNCTIONS</entry>
+<entry>?</entry>
+<entry>?</entry>
+</row>
+<row>
+<entry>SHOW DATABASES, SHOW SCHEMAS</entry>
+<entry>?</entry>
+<entry>?</entry>
+</row>
+</tbody>
+</tgroup>
+</table>
+-->
+
+<!-- Here are pieces from the original Hive table that do not apply to Impala because of differences in the SQL syntax.
+   Might resurrect some of these later as features are added to Impala, in particular CREATE TABLE AS SELECT.
+<row>
+<entry>EXPORT</entry>
+<entry>SELECT, INSERT@URI</entry>
+<entry>Table</entry>
+</row>
+<row>
+<entry>IMPORT</entry>
+<entry>ALL, SELECT@URI</entry>
+<entry>Database</entry>
+</row>
+<row>
+<entry>ANALYZE</entry>
+<entry>INSERT, SELECT</entry>
+<entry>Table</entry>
+</row>
+<row>
+<entry>SWITCH DATABASE</entry>
+<entry>Any</entry>
+<entry>Any Table, View in the DB</entry>
+</row>
+<row>
+<entry>DESCRIBE FUNCTION&#42;</entry>
+<entry>Not Restricted</entry>
+</row>
+<row>
+<entry>MSCK</entry>
+<entry>ALL</entry>
+<entry>Database</entry>
+</row>
+<row>
+<entry>SHOW COLUMNS</entry>
+<entry>SELECT or INSERT</entry>
+<entry>Table</entry>
+</row>
+<row>
+<entry>SHOW TABLE STATUS</entry>
+<entry>SELECT or INSERT</entry>
+<entry>Table</entry>
+</row>
+<row>
+<entry>SHOW TABLE PROPERTIES</entry>
+<entry>SELECT or INSERT</entry>
+<entry>Table</entry>
+</row>
+<row>
+<entry>SHOW CREATE TABLE</entry>
+<entry>SELECT or INSERT</entry>
+<entry>Table</entry>
+</row>
+<row>
+<entry>SHOW FUNCTIONS&#42;</entry>
+<entry>Not restricted</entry>
+</row>
+<row>
+<entry>SHOW PARTITIONS</entry>
+<entry>SELECT or INSERT</entry>
+<entry>Table</entry>
+</row>
+<row>
+<entry>SHOW INDEXES</entry>
+<entry>SELECT or INSERT</entry>
+<entry>Table</entry>
+</row>
+<row>
+<entry>SHOW LOCKS</entry>
+<entry>Not Restricted</entry>
+</row>
+<row>
+<entry>CREATE FUNCTION</entry>
+<entry>Any Privilege</entry>
+<entry>Any Object</entry>
+</row>
+<row>
+<entry>CREATE INDEX</entry>
+<entry>ALL</entry>
+<entry>Database</entry>
+</row>
+<row>
+<entry>DROP FUNCTION</entry>
+<entry>Any Privilege</entry>
+<entry>Any Object</entry>
+</row>
+<row>
+<entry>DROP INDEX</entry>
+<entry>ALL</entry>
+<entry>Database</entry>
+</row>
+<row>
+<entry>ALTER INDEX REBUILD</entry>
+<entry>ALL</entry>
+<entry>Database</entry>
+</row>
+<row>
+<entry>ALTER VIEW PROPERTIES</entry>
+<entry>ALL</entry>
+<entry>Database</entry>
+</row>
+<row>
+<entry>GRANT PRIVILEGE</entry>
+<entry>Allowed, but has no effect on HS2 auth</entry>
+</row>
+<row>
+<entry>REVOKE PRIVILEGE</entry>
+<entry>Allowed, but has no effect on HS2 auth</entry>
+</row>
+<row>
+<entry>SHOW GRANTS</entry>
+<entry>Allowed, but has no effect on HS2 auth</entry>
+</row>
+<row>
+<entry>ALTER TABLE PROTECT MODE</entry>
+<entry>ALL</entry>
+<entry>Database</entry>
+</row>
+<row>
+<entry>ALTER TABLE FILE FORMAT</entry>
+<entry>ALL</entry>
+<entry>Database</entry>
+</row>
+<row>
+<entry>ALTER PARTITION PROTECT MODE</entry>
+<entry>ALL</entry>
+<entry>Database</entry>
+</row>
+<row>
+<entry>ALTER PARTITION FILE FORMAT</entry>
+<entry>ALL</entry>
+<entry>Database</entry>
+</row>
+<row>
+<entry>ALTER PARTITION LOCATION&#42;</entry>
+<entry>ALL</entry>
+<entry>Server</entry>
+</row>
+<row>
+<entry>CREATE TABLE AS SELECT</entry>
+<entry>ALL, SELECT</entry>
+<entry>Database, Table/View</entry>
+</row>
+<row>
+<entry>ALTER INDEX PROP</entry>
+<entry>ALL</entry>
+<entry>Database</entry>
+</row>
+<row>
+<entry>ALTER DATABASE</entry>
+<entry>ALL</entry>
+<entry>Database</entry>
+</row>
+<row>
+<entry>DESCRIBE DATABASE</entry>
+<entry>ALL</entry>
+<entry>Database</entry>
+</row>
+<row>
+<entry>ALTER TABLE MERGE FILE</entry>
+<entry>ALL</entry>
+<entry>Database</entry>
+</row>
+<row>
+<entry>ALTER PARTITION MERGE FILE</entry>
+<entry>ALL</entry>
+<entry>Database</entry>
+</row>
+<row>
+<entry>ALTER TABLE SKEWED</entry>
+<entry>ALL</entry>
+<entry>Database</entry>
+</row>
+<row>
+<entry>ALTER TABLE PARTITION SKEWED LOCATION</entry>
+<entry>ALL</entry>
+<entry>Database</entry>
+</row>
+<row>
+<entry>ADD JAR&#42;</entry>
+<entry>Restricted unless hive.server2.authorization.external.exec = true</entry>
+</row>
+<row>
+<entry>TRANSFORM&#42;</entry>
+<entry>ALL</entry>
+<entry>Server</entry>
+</row>
+-->
+    </conbody>
+  </concept>
+
+  <concept id="sentry_debug">
+
+    <title><ph conref="../shared/impala_common.xml#common/title_sentry_debug"/></title>
+
+    <conbody>
+
+      <p conref="../shared/impala_common.xml#common/sentry_debug"/>
+    </conbody>
+  </concept>
+
+  <concept id="sentry_cm">
+
+    <title>Managing Sentry for Impala through Cloudera Manager</title>
+
+    <conbody>
+
+      <p>
+        To enable the Sentry service for Impala and Hive, set <uicontrol>Hive/Impala &gt; Service-Wide &gt; Sentry
+        Service</uicontrol> parameter to the Sentry service. Then restart Impala and Hive. Simply adding Sentry
+        service as a dependency and restarting enables Impala and Hive to use the Sentry service.
+      </p>
+
+      <p>
+        To set the server name to use when granting server level privileges, set the <uicontrol>Hive &gt;
+        Service-Wide &gt; Advanced &gt; Server Name for Sentry Authorization</uicontrol> parameter. When using
+        Sentry with the Hive Metastore, you can specify the list of users that are allowed to bypass Sentry
+        Authorization in Hive Metastore using <uicontrol>Hive &gt; Service-Wide &gt; Security &gt; Bypass Sentry
+        Authorization Users</uicontrol>. These are usually service users that already ensure all activity has been
+        authorized.
+      </p>
+
+      <note>
+        The <uicontrol>Hive/Impala &gt; Service-Wide &gt; Policy File Based Sentry</uicontrol> tab contains
+        parameters only relevant to configuring Sentry using policy files. In particular, make sure that
+        <uicontrol>Enable Sentry Authorization using Policy Files</uicontrol> parameter is unchecked when using the
+        Sentry service. Cloudera Manager throws a validation error if you attempt to configure the Sentry service
+        and policy file at the same time.
+      </note>
+    </conbody>
+  </concept>
+
+  <concept id="sec_ex_default">
+
+    <title>The DEFAULT Database in a Secure Deployment</title>
+
+    <conbody>
+
+      <p>
+        Because of the extra emphasis on granular access controls in a secure deployment, you should move any
+        important or sensitive information out of the <codeph>DEFAULT</codeph> database into a named database whose
+        privileges are specified in the policy file. Sometimes you might need to give privileges on the
+        <codeph>DEFAULT</codeph> database for administrative reasons; for example, as a place you can reliably
+        specify with a <codeph>USE</codeph> statement when preparing to drop a database.
+      </p>
+
+<!-- Maybe have an example later, but not for initial 1.1 release.
+<codeblock></codeblock>
+-->
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_avg.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_avg.xml b/docs/topics/impala_avg.xml
new file mode 100644
index 0000000..d95b2d5
--- /dev/null
+++ b/docs/topics/impala_avg.xml
@@ -0,0 +1,225 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="avg">
+
+  <title>AVG Function</title>
+  <titlealts audience="PDF"><navtitle>AVG</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="Analytic Functions"/>
+      <data name="Category" value="Aggregate Functions"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">avg() function</indexterm>
+      An aggregate function that returns the average value from a set of numbers or <codeph>TIMESTAMP</codeph> values.
+      Its single argument can be numeric column, or the numeric result of a function or expression applied to the
+      column value. Rows with a <codeph>NULL</codeph> value for the specified column are ignored. If the table is empty,
+      or all the values supplied to <codeph>AVG</codeph> are <codeph>NULL</codeph>, <codeph>AVG</codeph> returns
+      <codeph>NULL</codeph>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>AVG([DISTINCT | ALL] <varname>expression</varname>) [OVER (<varname>analytic_clause</varname>)]
+</codeblock>
+
+    <p>
+      When the query contains a <codeph>GROUP BY</codeph> clause, returns one value for each combination of
+      grouping values.
+    </p>
+
+    <p>
+      <b>Return type:</b> <codeph>DOUBLE</codeph> for numeric values; <codeph>TIMESTAMP</codeph> for
+      <codeph>TIMESTAMP</codeph> values
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_aggregation_explanation"/>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_aggregation_example"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>-- Average all the non-NULL values in a column.
+insert overwrite avg_t values (2),(4),(6),(null),(null);
+-- The average of the above values is 4: (2+4+6) / 3. The 2 NULL values are ignored.
+select avg(x) from avg_t;
+-- Average only certain values from the column.
+select avg(x) from t1 where month = 'January' and year = '2013';
+-- Apply a calculation to the value of the column before averaging.
+select avg(x/3) from t1;
+-- Apply a function to the value of the column before averaging.
+-- Here we are substituting a value of 0 for all NULLs in the column,
+-- so that those rows do factor into the return value.
+select avg(isnull(x,0)) from t1;
+-- Apply some number-returning function to a string column and average the results.
+-- If column s contains any NULLs, length(s) also returns NULL and those rows are ignored.
+select avg(length(s)) from t1;
+-- Can also be used in combination with DISTINCT and/or GROUP BY.
+-- Return more than one result.
+select month, year, avg(page_visits) from web_stats group by month, year;
+-- Filter the input to eliminate duplicates before performing the calculation.
+select avg(distinct x) from t1;
+-- Filter the output after performing the calculation.
+select avg(x) from t1 group by y having avg(x) between 1 and 20;
+</codeblock>
+
+    <p rev="2.0.0">
+      The following examples show how to use <codeph>AVG()</codeph> in an analytic context. They use a table
+      containing integers from 1 to 10. Notice how the <codeph>AVG()</codeph> is reported for each input value, as
+      opposed to the <codeph>GROUP BY</codeph> clause which condenses the result set.
+<codeblock>select x, property, avg(x) over (partition by property) as avg from int_t where property in ('odd','even');
++----+----------+-----+
+| x  | property | avg |
++----+----------+-----+
+| 2  | even     | 6   |
+| 4  | even     | 6   |
+| 6  | even     | 6   |
+| 8  | even     | 6   |
+| 10 | even     | 6   |
+| 1  | odd      | 5   |
+| 3  | odd      | 5   |
+| 5  | odd      | 5   |
+| 7  | odd      | 5   |
+| 9  | odd      | 5   |
++----+----------+-----+
+</codeblock>
+
+Adding an <codeph>ORDER BY</codeph> clause lets you experiment with results that are cumulative or apply to a moving
+set of rows (the <q>window</q>). The following examples use <codeph>AVG()</codeph> in an analytic context
+(that is, with an <codeph>OVER()</codeph> clause) to produce a running average of all the even values,
+then a running average of all the odd values. The basic <codeph>ORDER BY x</codeph> clause implicitly
+activates a window clause of <codeph>RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW</codeph>,
+which is effectively the same as <codeph>ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW</codeph>,
+therefore all of these examples produce the same results:
+<codeblock>select x, property,
+  avg(x) over (partition by property <b>order by x</b>) as 'cumulative average'
+  from int_t where property in ('odd','even');
++----+----------+--------------------+
+| x  | property | cumulative average |
++----+----------+--------------------+
+| 2  | even     | 2                  |
+| 4  | even     | 3                  |
+| 6  | even     | 4                  |
+| 8  | even     | 5                  |
+| 10 | even     | 6                  |
+| 1  | odd      | 1                  |
+| 3  | odd      | 2                  |
+| 5  | odd      | 3                  |
+| 7  | odd      | 4                  |
+| 9  | odd      | 5                  |
++----+----------+--------------------+
+
+select x, property,
+  avg(x) over
+  (
+    partition by property
+    <b>order by x</b>
+    <b>range between unbounded preceding and current row</b>
+  ) as 'cumulative average'
+from int_t where property in ('odd','even');
++----+----------+--------------------+
+| x  | property | cumulative average |
++----+----------+--------------------+
+| 2  | even     | 2                  |
+| 4  | even     | 3                  |
+| 6  | even     | 4                  |
+| 8  | even     | 5                  |
+| 10 | even     | 6                  |
+| 1  | odd      | 1                  |
+| 3  | odd      | 2                  |
+| 5  | odd      | 3                  |
+| 7  | odd      | 4                  |
+| 9  | odd      | 5                  |
++----+----------+--------------------+
+
+select x, property,
+  avg(x) over
+  (
+    partition by property
+    <b>order by x</b>
+    <b>rows between unbounded preceding and current row</b>
+  ) as 'cumulative average'
+  from int_t where property in ('odd','even');
++----+----------+--------------------+
+| x  | property | cumulative average |
++----+----------+--------------------+
+| 2  | even     | 2                  |
+| 4  | even     | 3                  |
+| 6  | even     | 4                  |
+| 8  | even     | 5                  |
+| 10 | even     | 6                  |
+| 1  | odd      | 1                  |
+| 3  | odd      | 2                  |
+| 5  | odd      | 3                  |
+| 7  | odd      | 4                  |
+| 9  | odd      | 5                  |
++----+----------+--------------------+
+</codeblock>
+
+The following examples show how to construct a moving window, with a running average taking into account 1 row before
+and 1 row after the current row, within the same partition (all the even values or all the odd values).
+Because of a restriction in the Impala <codeph>RANGE</codeph> syntax, this type of
+moving window is possible with the <codeph>ROWS BETWEEN</codeph> clause but not the <codeph>RANGE BETWEEN</codeph>
+clause:
+<codeblock>select x, property,
+  avg(x) over
+  (
+    partition by property
+    <b>order by x</b>
+    <b>rows between 1 preceding and 1 following</b>
+  ) as 'moving average'
+  from int_t where property in ('odd','even');
++----+----------+----------------+
+| x  | property | moving average |
++----+----------+----------------+
+| 2  | even     | 3              |
+| 4  | even     | 4              |
+| 6  | even     | 6              |
+| 8  | even     | 8              |
+| 10 | even     | 9              |
+| 1  | odd      | 2              |
+| 3  | odd      | 3              |
+| 5  | odd      | 5              |
+| 7  | odd      | 7              |
+| 9  | odd      | 8              |
++----+----------+----------------+
+
+-- Doesn't work because of syntax restriction on RANGE clause.
+select x, property,
+  avg(x) over
+  (
+    partition by property
+    <b>order by x</b>
+    <b>range between 1 preceding and 1 following</b>
+  ) as 'moving average'
+from int_t where property in ('odd','even');
+ERROR: AnalysisException: RANGE is only supported with both the lower and upper bounds UNBOUNDED or one UNBOUNDED and the other CURRENT ROW.
+</codeblock>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+<!-- This conref appears under SUM(), AVG(), FLOAT, and DOUBLE topics. -->
+
+    <p conref="../shared/impala_common.xml#common/sum_double"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_analytic_functions.xml#analytic_functions"/>, <xref href="impala_max.xml#max"/>,
+      <xref href="impala_min.xml#min"/>
+    </p>
+  </conbody>
+</concept>


[44/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_array.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_array.xml b/docs/topics/impala_array.xml
new file mode 100644
index 0000000..f9519c1
--- /dev/null
+++ b/docs/topics/impala_array.xml
@@ -0,0 +1,269 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="array">
+
+  <title>ARRAY Complex Type (<keyword keyref="impala23"/> or higher only)</title>
+
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      A complex data type that can represent an arbitrary number of ordered elements.
+      The elements can be scalars or another complex type (<codeph>ARRAY</codeph>,
+      <codeph>STRUCT</codeph>, or <codeph>MAP</codeph>).
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<!-- To do: make sure there is sufficient syntax info under the SELECT statement to understand how to query all the complex types. -->
+
+<codeblock><varname>column_name</varname> ARRAY &lt; <varname>type</varname> &gt;
+
+type ::= <varname>primitive_type</varname> | <varname>complex_type</varname>
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_combo"/>
+
+      <p>
+        The elements of the array have no names. You refer to the value of the array item using the
+        <codeph>ITEM</codeph> pseudocolumn, or its position in the array with the <codeph>POS</codeph>
+        pseudocolumn. See <xref href="impala_complex_types.xml#item"/> for information about
+        these pseudocolumns.
+      </p>
+
+<!-- Array is a frequently used idiom; don't recommend MAP right up front, since that is more rarely used. STRUCT has all different considerations.
+      <p>
+        If it would be logical to have a fixed number of elements and give each one a name, consider using a
+        <codeph>MAP</codeph> (when all elements are of the same type) or a <codeph>STRUCT</codeph> (if different
+        elements have different types) instead of an <codeph>ARRAY</codeph>.
+      </p>
+-->
+
+    <p>
+      Each row can have a different number of elements (including none) in the array for that row.
+    </p>
+
+<!-- Since you don't use numeric indexes, this assertion and advice doesn't make sense.
+      <p>
+        If you attempt to refer to a non-existent array element, the result is <codeph>NULL</codeph>. Therefore,
+        when using operations such as addition or string concatenation involving array elements, you might use
+        conditional functions to substitute default values such as 0 or <codeph>""</codeph> in the place of missing
+        array elements.
+      </p>
+-->
+
+      <p>
+        When an array contains items of scalar types, you can use aggregation functions on the array elements without using join notation. For
+        example, you can find the <codeph>COUNT()</codeph>, <codeph>AVG()</codeph>, <codeph>SUM()</codeph>, and so on of numeric array
+        elements, or the <codeph>MAX()</codeph> and <codeph>MIN()</codeph> of any scalar array elements by referring to
+        <codeph><varname>table_name</varname>.<varname>array_column</varname></codeph> in the <codeph>FROM</codeph> clause of the query. When
+        you need to cross-reference values from the array with scalar values from the same row, such as by including a <codeph>GROUP
+        BY</codeph> clause to produce a separate aggregated result for each row, then the join clause is required.
+      </p>
+
+      <p>
+        A common usage pattern with complex types is to have an array as the top-level type for the column:
+        an array of structs, an array of maps, or an array of arrays.
+        For example, you can model a denormalized table by creating a column that is an <codeph>ARRAY</codeph>
+        of <codeph>STRUCT</codeph> elements; each item in the array represents a row from a table that would
+        normally be used in a join query. This kind of data structure lets you essentially denormalize tables by
+        associating multiple rows from one table with the matching row in another table.
+      </p>
+
+      <p>
+        You typically do not create more than one top-level <codeph>ARRAY</codeph> column, because if there is
+        some relationship between the elements of multiple arrays, it is convenient to model the data as
+        an array of another complex type element (either <codeph>STRUCT</codeph> or <codeph>MAP</codeph>).
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_describe"/>
+
+      <p conref="../shared/impala_common.xml#common/added_in_230"/>
+
+      <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+      <ul conref="../shared/impala_common.xml#common/complex_types_restrictions">
+        <li/>
+      </ul>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <note conref="../shared/impala_common.xml#common/complex_type_schema_pointer"/>
+
+      <p>
+        The following example shows how to construct a table with various kinds of <codeph>ARRAY</codeph> columns,
+        both at the top level and nested within other complex types.
+        Whenever the <codeph>ARRAY</codeph> consists of a scalar value, such as in the <codeph>PETS</codeph>
+        column or the <codeph>CHILDREN</codeph> field, you can see that future expansion is limited.
+        For example, you could not easily evolve the schema to record the kind of pet or the child's birthday alongside the name.
+        Therefore, it is more common to use an <codeph>ARRAY</codeph> whose elements are of <codeph>STRUCT</codeph> type,
+        to associate multiple fields with each array element.
+      </p>
+
+      <note>
+        Practice the <codeph>CREATE TABLE</codeph> and query notation for complex type columns
+        using empty tables, until you can visualize a complex data structure and construct corresponding SQL statements reliably.
+      </note>
+
+<!-- To do: verify and flesh out this example. -->
+
+<codeblock><![CDATA[CREATE TABLE array_demo
+(
+  id BIGINT,
+  name STRING,
+-- An ARRAY of scalar type as a top-level column.
+  pets ARRAY <STRING>,
+
+-- An ARRAY with elements of complex type (STRUCT).
+  places_lived ARRAY < STRUCT <
+    place: STRING,
+    start_year: INT
+  >>,
+
+-- An ARRAY as a field (CHILDREN) within a STRUCT.
+-- (The STRUCT is inside another ARRAY, because it is rare
+-- for a STRUCT to be a top-level column.)
+  marriages ARRAY < STRUCT <
+    spouse: STRING,
+    children: ARRAY <STRING>
+  >>,
+
+-- An ARRAY as the value part of a MAP.
+-- The first MAP field (the key) would be a value such as
+-- 'Parent' or 'Grandparent', and the corresponding array would
+-- represent 2 parents, 4 grandparents, and so on.
+  ancestors MAP < STRING, ARRAY <STRING> >
+)
+STORED AS PARQUET;
+]]>
+</codeblock>
+
+    <p>
+      The following example shows how to examine the structure of a table containing one or more <codeph>ARRAY</codeph> columns by using the
+      <codeph>DESCRIBE</codeph> statement. You can visualize each <codeph>ARRAY</codeph> as its own two-column table, with columns
+      <codeph>ITEM</codeph> and <codeph>POS</codeph>.
+    </p>
+
+<!-- To do: extend the examples to include MARRIAGES and ANCESTORS columns, or get rid of those columns. -->
+
+<codeblock><![CDATA[DESCRIBE array_demo;
++--------------+---------------------------+
+| name         | type                      |
++--------------+---------------------------+
+| id           | bigint                    |
+| name         | string                    |
+| pets         | array<string>             |
+| marriages    | array<struct<             |
+|              |   spouse:string,          |
+|              |   children:array<string>  |
+|              | >>                        |
+| places_lived | array<struct<             |
+|              |   place:string,           |
+|              |   start_year:int          |
+|              | >>                        |
+| ancestors    | map<string,array<string>> |
++--------------+---------------------------+
+
+DESCRIBE array_demo.pets;
++------+--------+
+| name | type   |
++------+--------+
+| item | string |
+| pos  | bigint |
++------+--------+
+
+DESCRIBE array_demo.marriages;
++------+--------------------------+
+| name | type                     |
++------+--------------------------+
+| item | struct<                  |
+|      |   spouse:string,         |
+|      |   children:array<string> |
+|      | >                        |
+| pos  | bigint                   |
++------+--------------------------+
+
+DESCRIBE array_demo.places_lived;
++------+------------------+
+| name | type             |
++------+------------------+
+| item | struct<          |
+|      |   place:string,  |
+|      |   start_year:int |
+|      | >                |
+| pos  | bigint           |
++------+------------------+
+
+DESCRIBE array_demo.ancestors;
++-------+---------------+
+| name  | type          |
++-------+---------------+
+| key   | string        |
+| value | array<string> |
++-------+---------------+
+]]>
+</codeblock>
+
+    <p>
+      The following example shows queries involving <codeph>ARRAY</codeph> columns containing elements of scalar or complex types. You
+      <q>unpack</q> each <codeph>ARRAY</codeph> column by referring to it in a join query, as if it were a separate table with
+      <codeph>ITEM</codeph> and <codeph>POS</codeph> columns. If the array element is a scalar type, you refer to its value using the
+      <codeph>ITEM</codeph> pseudocolumn. If the array element is a <codeph>STRUCT</codeph>, you refer to the <codeph>STRUCT</codeph> fields
+      using dot notation and the field names. If the array element is another <codeph>ARRAY</codeph> or a <codeph>MAP</codeph>, you use
+      another level of join to unpack the nested collection elements.
+    </p>
+
+<!-- To do: have some sample output to show for these queries. -->
+
+<codeblock><![CDATA[-- Array of scalar values.
+-- Each array element represents a single string, plus we know its position in the array.
+SELECT id, name, pets.pos, pets.item FROM array_demo, array_demo.pets;
+
+-- Array of structs.
+-- Now each array element has named fields, possibly of different types.
+-- You can consider an ARRAY of STRUCT to represent a table inside another table.
+SELECT id, name, places_lived.pos, places_lived.item.place, places_lived.item.start_year
+FROM array_demo, array_demo.places_lived;
+
+-- The .ITEM name is optional for array elements that are structs.
+-- The following query is equivalent to the previous one, with .ITEM
+-- removed from the column references.
+SELECT id, name, places_lived.pos, places_lived.place, places_lived.start_year
+  FROM array_demo, array_demo.places_lived;
+
+-- To filter specific items from the array, do comparisons against the .POS or .ITEM
+-- pseudocolumns, or names of struct fields, in the WHERE clause.
+SELECT id, name, pets.item FROM array_demo, array_demo.pets
+  WHERE pets.pos in (0, 1, 3);
+
+SELECT id, name, pets.item FROM array_demo, array_demo.pets
+  WHERE pets.item LIKE 'Mr. %';
+
+SELECT id, name, places_lived.pos, places_lived.place, places_lived.start_year
+  FROM array_demo, array_demo.places_lived
+WHERE places_lived.place like '%California%';
+]]>
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_complex_types.xml#complex_types"/>,
+<!-- <xref href="impala_array.xml#array"/>, -->
+      <xref href="impala_struct.xml#struct"/>, <xref href="impala_map.xml#map"/>
+    </p>
+
+  </conbody>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_auditing.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_auditing.xml b/docs/topics/impala_auditing.xml
new file mode 100644
index 0000000..6332957
--- /dev/null
+++ b/docs/topics/impala_auditing.xml
@@ -0,0 +1,260 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="auditing">
+
+  <title>Auditing Impala Operations</title>
+  <titlealts audience="PDF"><navtitle>Auditing</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Auditing"/>
+      <data name="Category" value="Governance"/>
+      <data name="Category" value="Navigator"/>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="Administrators"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      To monitor how Impala data is being used within your organization, ensure that your Impala authorization and
+      authentication policies are effective, and detect attempts at intrusion or unauthorized access to Impala
+      data, you can use the auditing feature in Impala 1.2.1 and higher:
+    </p>
+
+    <ul>
+      <li>
+        Enable auditing by including the option <codeph>-audit_event_log_dir=<varname>directory_path</varname></codeph>
+        in your <cmdname>impalad</cmdname> startup options for a cluster not managed by Cloudera Manager, or
+        <xref audience="integrated" href="cn_iu_audit_log.xml#xd_583c10bfdbd326ba--6eed2fb8-14349d04bee--7d6f/section_v25_lmy_bn">configuring Impala Daemon logging in Cloudera Manager</xref><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/cn_iu_service_audit.html" scope="external" format="html">configuring Impala Daemon logging in Cloudera Manager</xref>.
+        The log directory must be a local directory on the
+        server, not an HDFS directory.
+      </li>
+
+      <li>
+        Decide how many queries will be represented in each log files. By default, Impala starts a new log file
+        every 5000 queries. To specify a different number, <ph
+          audience="standalone"
+          >include
+        the option <codeph>-max_audit_event_log_file_size=<varname>number_of_queries</varname></codeph> in the
+        <cmdname>impalad</cmdname> startup
+        options</ph><xref
+          href="cn_iu_audit_log.xml#xd_583c10bfdbd326ba--6eed2fb8-14349d04bee--7d6f/section_v25_lmy_bn"
+            audience="integrated"
+            >configure
+        Impala Daemon logging in Cloudera Manager</xref>.
+      </li>
+
+      <li> Configure Cloudera Navigator to collect and consolidate the audit
+        logs from all the hosts in the cluster. </li>
+
+      <li>
+        Use Cloudera Navigator or Cloudera Manager to filter, visualize, and produce reports based on the audit
+        data. (The Impala auditing feature works with Cloudera Manager 4.7 to 5.1 and Cloudera Navigator 2.1 and
+        higher.) Check the audit data to ensure that all activity is authorized and detect attempts at
+        unauthorized access.
+      </li>
+    </ul>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="auditing_performance">
+
+    <title>Durability and Performance Considerations for Impala Auditing</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Performance"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        The auditing feature only imposes performance overhead while auditing is enabled.
+      </p>
+
+      <p>
+        Because any Impala host can process a query, enable auditing on all hosts where the
+        <ph audience="standalone"><cmdname>impalad</cmdname> daemon</ph>
+        <ph audience="integrated">Impala Daemon role</ph> runs. Each host stores its own log
+        files, in a directory in the local filesystem. The log data is periodically flushed to disk (through an
+        <codeph>fsync()</codeph> system call) to avoid loss of audit data in case of a crash.
+      </p>
+
+      <p> The runtime overhead of auditing applies to whichever host serves as the coordinator for the query, that is, the host you connect to when you issue the query. This might be the same host for all queries, or different applications or users might connect to and issue queries through different hosts. </p>
+
+      <p> To avoid excessive I/O overhead on busy coordinator hosts, Impala syncs the audit log data (using the <codeph>fsync()</codeph> system call) periodically rather than after every query. Currently, the <codeph>fsync()</codeph> calls are issued at a fixed interval, every 5 seconds. </p>
+
+      <p>
+        By default, Impala avoids losing any audit log data in the case of an error during a logging operation
+        (such as a disk full error), by immediately shutting down
+        <cmdname audience="standalone">impalad</cmdname><ph audience="integrated">the Impala
+        Daemon role</ph> on the host where the auditing problem occurred.
+        <ph audience="standalone">You can override this setting by specifying the option
+        <codeph>-abort_on_failed_audit_event=false</codeph> in the <cmdname>impalad</cmdname> startup options.</ph>
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="auditing_format">
+
+    <title>Format of the Audit Log Files</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Logs"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p> The audit log files represent the query information in JSON format, one query per line. Typically, rather than looking at the log files themselves, you use the Cloudera Navigator product to consolidate the log data from all Impala hosts and filter and visualize the results in useful ways. (If you do examine the raw log data, you might run the files through a JSON pretty-printer first.) </p>
+
+      <p>
+        All the information about schema objects accessed by the query is encoded in a single nested record on the
+        same line. For example, the audit log for an <codeph>INSERT ... SELECT</codeph> statement records that a
+        select operation occurs on the source table and an insert operation occurs on the destination table. The
+        audit log for a query against a view records the base table accessed by the view, or multiple base tables
+        in the case of a view that includes a join query. Every Impala operation that corresponds to a SQL
+        statement is recorded in the audit logs, whether the operation succeeds or fails. Impala records more
+        information for a successful operation than for a failed one, because an unauthorized query is stopped
+        immediately, before all the query planning is completed.
+      </p>
+
+<!-- Opportunity to conref at the phrase level here... the content of this paragraph is the same as part
+     of a list bullet earlier on. -->
+
+      <p>
+        The information logged for each query includes:
+      </p>
+
+      <ul>
+        <li>
+          Client session state:
+          <ul>
+            <li>
+              Session ID
+            </li>
+
+            <li>
+              User name
+            </li>
+
+            <li>
+              Network address of the client connection
+            </li>
+          </ul>
+        </li>
+
+        <li>
+          SQL statement details:
+          <ul>
+            <li>
+              Query ID
+            </li>
+
+            <li>
+              Statement Type - DML, DDL, and so on
+            </li>
+
+            <li>
+              SQL statement text
+            </li>
+
+            <li>
+              Execution start time, in local time
+            </li>
+
+            <li>
+              Execution Status - Details on any errors that were encountered
+            </li>
+
+            <li>
+              Target Catalog Objects:
+              <ul>
+                <li>
+                  Object Type - Table, View, or Database
+                </li>
+
+                <li>
+                  Fully qualified object name
+                </li>
+
+                <li>
+                  Privilege - How the object is being used (<codeph>SELECT</codeph>, <codeph>INSERT</codeph>,
+                  <codeph>CREATE</codeph>, and so on)
+                </li>
+              </ul>
+            </li>
+          </ul>
+        </li>
+      </ul>
+
+<!-- Delegating actual examples to the Cloudera Navigator doc for the moment.
+<p>
+Here is an excerpt from a sample audit log file:
+</p>
+<codeblock></codeblock>
+-->
+    </conbody>
+  </concept>
+
+  <concept id="auditing_exceptions">
+
+    <title>Which Operations Are Audited</title>
+
+    <conbody>
+
+      <p>
+        The kinds of SQL queries represented in the audit log are:
+      </p>
+
+      <ul>
+        <li>
+          Queries that are prevented due to lack of authorization.
+        </li>
+
+        <li>
+          Queries that Impala can analyze and parse to determine that they are authorized. The audit data is
+          recorded immediately after Impala finishes its analysis, before the query is actually executed.
+        </li>
+      </ul>
+
+      <p>
+        The audit log does not contain entries for queries that could not be parsed and analyzed. For example, a
+        query that fails due to a syntax error is not recorded in the audit log. The audit log also does not
+        contain queries that fail due to a reference to a table that does not exist, if you would be authorized to
+        access the table if it did exist.
+      </p>
+
+      <p>
+        Certain statements in the <cmdname>impala-shell</cmdname> interpreter, such as <codeph>CONNECT</codeph>,
+        <codeph rev="1.4.0">SUMMARY</codeph>, <codeph>PROFILE</codeph>, <codeph>SET</codeph>, and
+        <codeph>QUIT</codeph>, do not correspond to actual SQL queries, and these statements are not reflected in
+        the audit log.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="auditing_reviewing">
+
+    <title>Reviewing the Audit Logs</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Logs"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        You typically do not review the audit logs in raw form. The Cloudera Manager Agent periodically transfers
+        the log information into a back-end database where it can be examined in consolidated form. See
+        <ph audience="standalone">the <xref href="http://www.cloudera.com/content/cloudera-content/cloudera-docs/Navigator/latest/Cloudera-Navigator-Installation-and-User-Guide/Cloudera-Navigator-Installation-and-User-Guide.html"
+            scope="external" format="html">Cloudera Navigator documentation</xref> for details</ph>
+            <xref href="cn_iu_audits.xml#cn_topic_7" audience="integrated" />.
+      </p>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_authentication.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_authentication.xml b/docs/topics/impala_authentication.xml
new file mode 100644
index 0000000..7200e5f
--- /dev/null
+++ b/docs/topics/impala_authentication.xml
@@ -0,0 +1,39 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="authentication">
+
+  <title>Impala Authentication</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Authentication"/>
+      <data name="Category" value="Administrators"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Authentication is the mechanism to ensure that only specified hosts and users can connect to Impala. It also
+      verifies that when clients connect to Impala, they are connected to a legitimate server. This feature
+      prevents spoofing such as <term>impersonation</term> (setting up a phony client system with the same account
+      and group names as a legitimate user) and <term>man-in-the-middle attacks</term> (intercepting application
+      requests before they reach Impala and eavesdropping on sensitive information in the requests or the results).
+    </p>
+
+    <p>
+      Impala supports authentication using either Kerberos or LDAP.
+    </p>
+
+    <note conref="../shared/impala_common.xml#common/authentication_vs_authorization"/>
+
+    <p outputclass="toc"/>
+
+    <p>
+      Once you are finished setting up authentication, move on to authorization, which involves specifying what
+      databases, tables, HDFS directories, and so on can be accessed by particular users when they connect through
+      Impala. See <xref href="impala_authorization.xml#authorization"/> for details.
+    </p>
+  </conbody>
+</concept>


[03/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_tutorial.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_tutorial.xml b/docs/topics/impala_tutorial.xml
new file mode 100644
index 0000000..c5c8505
--- /dev/null
+++ b/docs/topics/impala_tutorial.xml
@@ -0,0 +1,2500 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="tutorial">
+
+  <title id="tutorials">Impala Tutorials</title>
+  <titlealts audience="PDF"><navtitle>Tutorials</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Tutorials"/>
+      <data name="Category" value="Getting Started"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      This section includes tutorial scenarios that demonstrate how to begin using Impala once the software is
+      installed. It focuses on techniques for loading data, because once you have some data in tables and can query
+      that data, you can quickly progress to more advanced Impala features.
+    </p>
+
+    <note>
+      <p>
+        Where practical, the tutorials take you from <q>ground zero</q> to having the desired Impala tables and
+        data. In some cases, you might need to download additional files from outside sources, set up additional
+        software components, modify commands or scripts to fit your own configuration, or substitute your own
+        sample data.
+      </p>
+    </note>
+
+    <p>
+      Before trying these tutorial lessons, install Impala using one of these procedures:
+    </p>
+
+    <ul>
+      <li>
+        If you already have a CDH environment set up and just need to add Impala to it, follow the installation
+        process described in <xref href="impala_install.xml#install"/>. Make sure to also install the Hive
+        metastore service if you do not already have Hive configured.
+      </li>
+
+      <li>
+        To set up Impala and all its prerequisites at once, in a minimal configuration that you can use for
+        small-scale experiments, set up the Cloudera QuickStart VM, which includes CDH and Impala on CentOS.
+        Use this single-node VM to try out basic SQL functionality, not anything related to
+        performance and scalability. For more information, see
+        <xref href="http://www.cloudera.com/content/support/en/downloads/quickstart_vms.html" scope="external" format="html">the
+        Cloudera QuickStart VM</xref>.
+      </li>
+    </ul>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="tut_beginner">
+
+    <title>Tutorials for Getting Started</title>
+
+    <conbody>
+
+      <p>
+        These tutorials demonstrate the basics of using Impala. They are intended for first-time users, and for
+        trying out Impala on any new cluster to make sure the major components are working correctly.
+      </p>
+
+      <p outputclass="toc inpage"/>
+    </conbody>
+
+    <concept id="tutorial_explore">
+
+      <title>Explore a New Impala Instance</title>
+
+      <conbody>
+
+        <p>
+          This tutorial demonstrates techniques for finding your way around the tables and databases of an
+          unfamiliar (possibly empty) Impala instance.
+        </p>
+
+        <p>
+          When you connect to an Impala instance for the first time, you use the <codeph>SHOW DATABASES</codeph>
+          and <codeph>SHOW TABLES</codeph> statements to view the most common types of objects. Also, call the
+          <codeph>version()</codeph> function to confirm which version of Impala you are running; the version
+          number is important when consulting documentation and dealing with support issues.
+        </p>
+
+        <p>
+          A completely empty Impala instance contains no tables, but still has two databases:
+        </p>
+
+        <ul>
+          <li>
+            <codeph>default</codeph>, where new tables are created when you do not specify any other database.
+          </li>
+
+          <li>
+            <codeph>_impala_builtins</codeph>, a system database used to hold all the built-in functions.
+          </li>
+        </ul>
+
+        <p>
+          The following example shows how to see the available databases, and the tables in each. If the list of
+          databases or tables is long, you can use wildcard notation to locate specific databases or tables based
+          on their names.
+        </p>
+
+<codeblock>$ impala-shell -i localhost --quiet
+Starting Impala Shell without Kerberos authentication
+Welcome to the Impala shell. Press TAB twice to see a list of available commands.
+
+Copyright (c) 2012 Cloudera, Inc. All rights reserved.
+
+(Shell build version: Impala Shell v...
+[localhost:21000] > select version();
++-------------------------------------------
+| version()
++-------------------------------------------
+| impalad version ...
+| Built on ...
++-------------------------------------------
+[localhost:21000] > show databases;
++--------------------------+
+| name                     |
++--------------------------+
+| _impala_builtins         |
+| ctas                     |
+| d1                       |
+| d2                       |
+| d3                       |
+| default                  |
+| explain_plans            |
+| external_table           |
+| file_formats             |
+| tpc                      |
++--------------------------+
+[localhost:21000] > select current_database();
++--------------------+
+| current_database() |
++--------------------+
+| default            |
++--------------------+
+[localhost:21000] > show tables;
++-------+
+| name  |
++-------+
+| ex_t  |
+| t1    |
++-------+
+[localhost:21000] > show tables in d3;
+
+[localhost:21000] > show tables in tpc;
++------------------------+
+| name                   |
++------------------------+
+| city                   |
+| customer               |
+| customer_address       |
+| customer_demographics  |
+| household_demographics |
+| item                   |
+| promotion              |
+| store                  |
+| store2                 |
+| store_sales            |
+| ticket_view            |
+| time_dim               |
+| tpc_tables             |
++------------------------+
+[localhost:21000] > show tables in tpc like 'customer*';
++-----------------------+
+| name                  |
++-----------------------+
+| customer              |
+| customer_address      |
+| customer_demographics |
++-----------------------+
+</codeblock>
+
+        <p>
+          Once you know what tables and databases are available, you descend into a database with the
+          <codeph>USE</codeph> statement. To understand the structure of each table, you use the
+          <codeph>DESCRIBE</codeph> command. Once inside a database, you can issue statements such as
+          <codeph>INSERT</codeph> and <codeph>SELECT</codeph> that operate on particular tables.
+        </p>
+
+        <p>
+          The following example explores a database named <codeph>TPC</codeph> whose name we learned in the
+          previous example. It shows how to filter the table names within a database based on a search string,
+          examine the columns of a table, and run queries to examine the characteristics of the table data. For
+          example, for an unfamiliar table you might want to know the number of rows, the number of different
+          values for a column, and other properties such as whether the column contains any <codeph>NULL</codeph>
+          values. When sampling the actual data values from a table, use a <codeph>LIMIT</codeph> clause to avoid
+          excessive output if the table contains more rows or distinct values than you expect.
+        </p>
+
+<codeblock>[localhost:21000] > use tpc;
+[localhost:21000] > show tables like '*view*';
++-------------+
+| name        |
++-------------+
+| ticket_view |
++-------------+
+[localhost:21000] > describe city;
++-------------+--------+---------+
+| name        | type   | comment |
++-------------+--------+---------+
+| id          | int    |         |
+| name        | string |         |
+| countrycode | string |         |
+| district    | string |         |
+| population  | int    |         |
++-------------+--------+---------+
+[localhost:21000] > select count(*) from city;
++----------+
+| count(*) |
++----------+
+| 0        |
++----------+
+[localhost:21000] > desc customer;
++------------------------+--------+---------+
+| name                   | type   | comment |
++------------------------+--------+---------+
+| c_customer_sk          | int    |         |
+| c_customer_id          | string |         |
+| c_current_cdemo_sk     | int    |         |
+| c_current_hdemo_sk     | int    |         |
+| c_current_addr_sk      | int    |         |
+| c_first_shipto_date_sk | int    |         |
+| c_first_sales_date_sk  | int    |         |
+| c_salutation           | string |         |
+| c_first_name           | string |         |
+| c_last_name            | string |         |
+| c_preferred_cust_flag  | string |         |
+| c_birth_day            | int    |         |
+| c_birth_month          | int    |         |
+| c_birth_year           | int    |         |
+| c_birth_country        | string |         |
+| c_login                | string |         |
+| c_email_address        | string |         |
+| c_last_review_date     | string |         |
++------------------------+--------+---------+
+[localhost:21000] > select count(*) from customer;
++----------+
+| count(*) |
++----------+
+| 100000   |
++----------+
+[localhost:21000] > select count(distinct c_birth_month) from customer;
++-------------------------------+
+| count(distinct c_birth_month) |
++-------------------------------+
+| 12                            |
++-------------------------------+
+[localhost:21000] > select count(*) from customer where c_email_address is null;
++----------+
+| count(*) |
++----------+
+| 0        |
++----------+
+[localhost:21000] > select distinct c_salutation from customer limit 10;
++--------------+
+| c_salutation |
++--------------+
+| Mr.          |
+| Ms.          |
+| Dr.          |
+|              |
+| Miss         |
+| Sir          |
+| Mrs.         |
++--------------+
+</codeblock>
+
+        <p>
+          When you graduate from read-only exploration, you use statements such as <codeph>CREATE DATABASE</codeph>
+          and <codeph>CREATE TABLE</codeph> to set up your own database objects.
+        </p>
+
+        <p>
+          The following example demonstrates creating a new database holding a new table. Although the last example
+          ended inside the <codeph>TPC</codeph> database, the new <codeph>EXPERIMENTS</codeph> database is not
+          nested inside <codeph>TPC</codeph>; all databases are arranged in a single top-level list.
+        </p>
+
+<codeblock>[localhost:21000] > create database experiments;
+[localhost:21000] > show databases;
++--------------------------+
+| name                     |
++--------------------------+
+| _impala_builtins         |
+| ctas                     |
+| d1                       |
+| d2                       |
+| d3                       |
+| default                  |
+| experiments              |
+| explain_plans            |
+| external_table           |
+| file_formats             |
+| tpc                      |
++--------------------------+
+[localhost:21000] > show databases like 'exp*';
++---------------+
+| name          |
++---------------+
+| experiments   |
+| explain_plans |
++---------------+
+</codeblock>
+
+        <p>
+          The following example creates a new table, <codeph>T1</codeph>. To illustrate a common mistake, it creates this table inside
+          the wrong database, the <codeph>TPC</codeph> database where the previous example ended. The <codeph>ALTER
+          TABLE</codeph> statement lets you move the table to the intended database, <codeph>EXPERIMENTS</codeph>, as part of a rename operation.
+          The <codeph>USE</codeph> statement is always needed to switch to a new database, and the
+          <codeph>current_database()</codeph> function confirms which database the session is in, to avoid these
+          kinds of mistakes.
+        </p>
+
+<codeblock>[localhost:21000] > create table t1 (x int);
+
+[localhost:21000] > show tables;
++------------------------+
+| name                   |
++------------------------+
+| city                   |
+| customer               |
+| customer_address       |
+| customer_demographics  |
+| household_demographics |
+| item                   |
+| promotion              |
+| store                  |
+| store2                 |
+| store_sales            |
+| t1                     |
+| ticket_view            |
+| time_dim               |
+| tpc_tables             |
++------------------------+
+[localhost:21000] > select current_database();
++--------------------+
+| current_database() |
++--------------------+
+| tpc                |
++--------------------+
+[localhost:21000] > alter table t1 rename to experiments.t1;
+[localhost:21000] > use experiments;
+[localhost:21000] > show tables;
++------+
+| name |
++------+
+| t1   |
++------+
+[localhost:21000] > select current_database();
++--------------------+
+| current_database() |
++--------------------+
+| experiments        |
++--------------------+
+</codeblock>
+
+        <p>
+          For your initial experiments with tables, you can use ones with just a few columns and a few rows, and
+          text-format data files.
+        </p>
+
+        <note>
+          As you graduate to more realistic scenarios, you will use more elaborate tables with many columns,
+          features such as partitioning, and file formats such as Parquet. When dealing with realistic data
+          volumes, you will bring in data using <codeph>LOAD DATA</codeph> or <codeph>INSERT ... SELECT</codeph>
+          statements to operate on millions or billions of rows at once.
+        </note>
+
+        <p>
+          The following example sets up a couple of simple tables with a few rows, and performs queries involving
+          sorting, aggregate functions and joins.
+        </p>
+
+<codeblock>[localhost:21000] > insert into t1 values (1), (3), (2), (4);
+[localhost:21000] > select x from t1 order by x desc;
++---+
+| x |
++---+
+| 4 |
+| 3 |
+| 2 |
+| 1 |
++---+
+[localhost:21000] > select min(x), max(x), sum(x), avg(x) from t1;
++--------+--------+--------+--------+
+| min(x) | max(x) | sum(x) | avg(x) |
++--------+--------+--------+--------+
+| 1      | 4      | 10     | 2.5    |
++--------+--------+--------+--------+
+
+[localhost:21000] > create table t2 (id int, word string);
+[localhost:21000] > insert into t2 values (1, "one"), (3, "three"), (5, 'five');
+[localhost:21000] > select word from t1 join t2 on (t1.x = t2.id);
++-------+
+| word  |
++-------+
+| one   |
+| three |
++-------+
+</codeblock>
+
+        <p>
+          After completing this tutorial, you should now know:
+        </p>
+
+        <ul>
+          <li>
+            How to tell which version of Impala is running on your system.
+          </li>
+
+          <li>
+            How to find the names of databases in an Impala instance, either displaying the full list or
+            searching for specific names.
+          </li>
+
+          <li>
+            How to find the names of tables in an Impala database, either displaying the full list or
+            searching for specific names.
+          </li>
+
+          <li>
+            How to switch between databases and check which database you are currently in.
+          </li>
+
+          <li>
+            How to learn the column names and types of a table.
+          </li>
+
+          <li>
+            How to create databases and tables, insert small amounts of test data, and run simple queries.
+          </li>
+        </ul>
+      </conbody>
+    </concept>
+
+    <concept audience="Cloudera" id="tutorial_quickstart_vm">
+
+      <title>Setting Up Impala with the QuickStart VM</title>
+
+      <conbody>
+
+        <p>
+          This tutorial scenario shows how to set up the Cloudera QuickStart VM, load some sample data, and begin
+          experimenting with Impala inside this virtualized environment.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept audience="Cloudera" id="tutorial_hdfs">
+
+      <title>Exploring the HDFS Directory Structure</title>
+
+      <conbody>
+
+        <p>
+          This tutorial scenario illustrates the HDFS directory structures that correspond to various
+          Impala databases, tables, and partitions. It also shows how data directories are shared between
+          Impala and Hive, because of the shared metastore database.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept audience="Cloudera" id="tutorial_external_table">
+
+      <title/>
+
+      <conbody>
+
+        <p>
+          In this tutorial scenario, you create a simple text-format data file in HDFS and then define an Impala
+          table that refers to the data in its original location.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept id="tutorial_csv_setup">
+
+      <title>Load CSV Data from Local Files</title>
+
+      <conbody>
+
+        <p>
+          This scenario illustrates how to create some very small tables, suitable for first-time users to
+          experiment with Impala SQL features. <codeph>TAB1</codeph> and <codeph>TAB2</codeph> are loaded with data
+          from files in HDFS. A subset of data is copied from <codeph>TAB1</codeph> into <codeph>TAB3</codeph>.
+        </p>
+
+        <p>
+          Populate HDFS with the data you want to query. To begin this process, create one or more new
+          subdirectories underneath your user directory in HDFS. The data for each table resides in a separate
+          subdirectory. Substitute your own username for <codeph>cloudera</codeph> where appropriate. This example
+          uses the <codeph>-p</codeph> option with the <codeph>mkdir</codeph> operation to create any necessary
+          parent directories if they do not already exist.
+        </p>
+
+<codeblock>$ whoami
+cloudera
+$ hdfs dfs -ls /user
+Found 3 items
+drwxr-xr-x   - cloudera cloudera            0 2013-04-22 18:54 /user/cloudera
+drwxrwx---   - mapred   mapred              0 2013-03-15 20:11 /user/history
+drwxr-xr-x   - hue      supergroup          0 2013-03-15 20:10 /user/hive
+<!-- $ hdfs dfs -mkdir -p /user/cloudera/sample_data/tab1 -->
+$ hdfs dfs -mkdir -p /user/cloudera/sample_data/tab1 /user/cloudera/sample_data/tab2</codeblock>
+
+        <p>
+          Here is some sample data, for two tables named <codeph>TAB1</codeph> and <codeph>TAB2</codeph>.
+        </p>
+
+        <p>
+          Copy the following content to <codeph>.csv</codeph> files in your local filesystem:
+        </p>
+
+        <p>
+          <filepath>tab1.csv</filepath>:
+        </p>
+
+<codeblock>1,true,123.123,2012-10-24 08:55:00
+2,false,1243.5,2012-10-25 13:40:00
+3,false,24453.325,2008-08-22 09:33:21.123
+4,false,243423.325,2007-05-12 22:32:21.33454
+5,true,243.325,1953-04-22 09:11:33
+</codeblock>
+
+        <p>
+          <filepath>tab2.csv</filepath>:
+        </p>
+
+<codeblock>1,true,12789.123
+2,false,1243.5
+3,false,24453.325
+4,false,2423.3254
+5,true,243.325
+60,false,243565423.325
+70,true,243.325
+80,false,243423.325
+90,true,243.325
+</codeblock>
+
+        <p>
+          Put each <codeph>.csv</codeph> file into a separate HDFS directory using commands like the following,
+          which use paths available in the Impala Demo VM:
+        </p>
+
+<codeblock><!-- $ hdfs dfs -mkdir /user/cloudera/tab1 -->$ hdfs dfs -put tab1.csv /user/cloudera/sample_data/tab1
+$ hdfs dfs -ls /user/cloudera/sample_data/tab1
+Found 1 items
+-rw-r--r--   1 cloudera cloudera        192 2013-04-02 20:08 /user/cloudera/sample_data/tab1/tab1.csv
+
+<!-- $ hdfs dfs -mkdir /user/cloudera/tab2 -->
+$ hdfs dfs -put tab2.csv /user/cloudera/sample_data/tab2
+$ hdfs dfs -ls /user/cloudera/sample_data/tab2
+Found 1 items
+-rw-r--r--   1 cloudera cloudera        158 2013-04-02 20:09 /user/cloudera/sample_data/tab2/tab2.csv
+</codeblock>
+
+        <p>
+          The name of each data file is not significant. In fact, when Impala examines the contents of the data
+          directory for the first time, it considers all files in the directory to make up the data of the table,
+          regardless of how many files there are or what the files are named.
+        </p>
+
+        <p>
+          To understand what paths are available within your own HDFS filesystem and what the permissions are for
+          the various directories and files, issue <codeph>hdfs dfs -ls /</codeph> and work your way down the tree
+          doing <codeph>-ls</codeph> operations for the various directories.
+        </p>
+
+        <p>
+          Use the <codeph>impala-shell</codeph> command to create tables, either interactively or through a SQL
+          script.
+        </p>
+
+        <p>
+          The following example shows creating three tables. For each table, the example shows creating columns
+          with various attributes such as Boolean or integer types. The example also includes commands that provide
+          information about how the data is formatted, such as rows terminating with commas, which makes sense in
+          the case of importing data from a <codeph>.csv</codeph> file. Where we already have <codeph>.csv</codeph>
+          files containing data in the HDFS directory tree, we specify the location of the directory containing the
+          appropriate <codeph>.csv</codeph> file. Impala considers all the data from all the files in that
+          directory to represent the data for the table.
+        </p>
+
+<codeblock>DROP TABLE IF EXISTS tab1;
+-- The EXTERNAL clause means the data is located outside the central location
+-- for Impala data files and is preserved when the associated Impala table is dropped.
+-- We expect the data to already exist in the directory specified by the LOCATION clause.
+CREATE EXTERNAL TABLE tab1
+(
+   id INT,
+   col_1 BOOLEAN,
+   col_2 DOUBLE,
+   col_3 TIMESTAMP
+)
+ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
+LOCATION '/user/cloudera/sample_data/tab1';
+
+DROP TABLE IF EXISTS tab2;
+-- TAB2 is an external table, similar to TAB1.
+CREATE EXTERNAL TABLE tab2
+(
+   id INT,
+   col_1 BOOLEAN,
+   col_2 DOUBLE
+)
+ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
+LOCATION '/user/cloudera/sample_data/tab2';
+
+DROP TABLE IF EXISTS tab3;
+-- Leaving out the EXTERNAL clause means the data will be managed
+-- in the central Impala data directory tree. Rather than reading
+-- existing data files when the table is created, we load the
+-- data after creating the table.
+CREATE TABLE tab3
+(
+   id INT,
+   col_1 BOOLEAN,
+   col_2 DOUBLE,
+   month INT,
+   day INT
+)
+ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
+</codeblock>
+
+        <note>
+          Getting through these <codeph>CREATE TABLE</codeph> statements successfully is an important validation
+          step to confirm everything is configured correctly with the Hive metastore and HDFS permissions. If you
+          receive any errors during the <codeph>CREATE TABLE</codeph> statements:
+          <ul>
+            <li>
+              Make sure you followed the installation instructions closely, in
+              <xref href="impala_install.xml#install"/>.
+            </li>
+
+            <li>
+              Make sure the <codeph>hive.metastore.warehouse.dir</codeph> property points to a directory that
+              Impala can write to. The ownership should be <codeph>hive:hive</codeph>, and the
+              <codeph>impala</codeph> user should also be a member of the <codeph>hive</codeph> group.
+            </li>
+
+            <li>
+              If the value of <codeph>hive.metastore.warehouse.dir</codeph> is different in the Cloudera Manager
+              dialogs and in the Hive shell, you might need to
+              <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_managing_roles.html" scope="external" format="html">designate
+              the hosts running <cmdname>impalad</cmdname> with the <q>gateway</q> role for Hive</xref>, and
+              <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_client_config.html" scope="external" format="html">deploy
+              the client configuration files</xref> to those hosts.
+            </li>
+          </ul>
+        </note>
+      </conbody>
+    </concept>
+
+    <concept id="tutorial_create_table">
+
+      <title>Point an Impala Table at Existing Data Files</title>
+
+      <conbody>
+
+        <p>
+          A convenient way to set up data for Impala to access is to use an external table, where the data already
+          exists in a set of HDFS files and you just point the Impala table at the directory containing those
+          files. For example, you might run in <codeph>impala-shell</codeph> a <codeph>*.sql</codeph> file with
+          contents similar to the following, to create an Impala table that accesses an existing data file used by
+          Hive.
+        </p>
+
+        <p>
+          The following examples set up 2 tables, referencing the paths and sample data supplied with the Cloudera
+          QuickStart VM. For historical reasons, the data physically resides in an HDFS directory tree under
+          <filepath>/user/hive</filepath>, although this particular data is entirely managed by Impala rather than
+          Hive. When we create an external table, we specify the directory containing one or more data files, and
+          Impala queries the combined content of all the files inside that directory. Here is how we examine the
+          directories and files within the HDFS filesystem:
+        </p>
+
+<codeblock>$ cd ~/cloudera/datasets
+$ ./tpcds-setup.sh
+... Downloads and unzips the kit, builds the data and loads it into HDFS ...
+$ hdfs dfs -ls /user/hive/tpcds/customer
+Found 1 items
+-rw-r--r--   1 cloudera supergroup   13209372 2013-03-22 18:09 /user/hive/tpcds/customer/customer.dat
+$ hdfs dfs -cat /user/hive/tpcds/customer/customer.dat | more
+1|AAAAAAAABAAAAAAA|980124|7135|32946|2452238|2452208|Mr.|Javier|Lewis|Y|9|12|1936|CHILE||Javie
+r.Lewis@VFAxlnZEvOx.org|2452508|
+2|AAAAAAAACAAAAAAA|819667|1461|31655|2452318|2452288|Dr.|Amy|Moses|Y|9|4|1966|TOGO||Amy.Moses@
+Ovk9KjHH.com|2452318|
+3|AAAAAAAADAAAAAAA|1473522|6247|48572|2449130|2449100|Miss|Latisha|Hamilton|N|18|9|1979|NIUE||
+Latisha.Hamilton@V.com|2452313|
+4|AAAAAAAAEAAAAAAA|1703214|3986|39558|2450030|2450000|Dr.|Michael|White|N|7|6|1983|MEXICO||Mic
+hael.White@i.org|2452361|
+5|AAAAAAAAFAAAAAAA|953372|4470|36368|2449438|2449408|Sir|Robert|Moran|N|8|5|1956|FIJI||Robert.
+Moran@Hh.edu|2452469|
+...
+</codeblock>
+
+        <p>
+          Here is a SQL script to set up Impala tables pointing to some of these data files in HDFS.
+          (The script in the VM sets up tables like this through Hive; ignore those tables
+          for purposes of this demonstration.)
+          Save the following as <filepath>customer_setup.sql</filepath>:
+        </p>
+
+<codeblock>--
+-- store_sales fact table and surrounding dimension tables only
+--
+create database tpcds;
+use tpcds;
+
+drop table if exists customer;
+create external table customer
+(
+    c_customer_sk             int,
+    c_customer_id             string,
+    c_current_cdemo_sk        int,
+    c_current_hdemo_sk        int,
+    c_current_addr_sk         int,
+    c_first_shipto_date_sk    int,
+    c_first_sales_date_sk     int,
+    c_salutation              string,
+    c_first_name              string,
+    c_last_name               string,
+    c_preferred_cust_flag     string,
+    c_birth_day               int,
+    c_birth_month             int,
+    c_birth_year              int,
+    c_birth_country           string,
+    c_login                   string,
+    c_email_address           string,
+    c_last_review_date        string
+)
+row format delimited fields terminated by '|'
+location '/user/hive/tpcds/customer';
+
+drop table if exists customer_address;
+create external table customer_address
+(
+    ca_address_sk             int,
+    ca_address_id             string,
+    ca_street_number          string,
+    ca_street_name            string,
+    ca_street_type            string,
+    ca_suite_number           string,
+    ca_city                   string,
+    ca_county                 string,
+    ca_state                  string,
+    ca_zip                    string,
+    ca_country                string,
+    ca_gmt_offset             float,
+    ca_location_type          string
+)
+row format delimited fields terminated by '|'
+location '/user/hive/tpcds/customer_address';
+</codeblock>
+
+        <p>
+          We would run this script with a command such as:
+<codeblock>impala-shell -i localhost -f customer_setup.sql</codeblock>
+        </p>
+
+      </conbody>
+    </concept>
+
+    <concept id="tutorial_describe_impala">
+
+      <title>Describe the Impala Table</title>
+
+      <conbody>
+
+        <p>
+          Now that you have updated the database metadata that Impala caches, you can confirm that the expected
+          tables are accessible by Impala and examine the attributes of one of the tables. We created these tables
+          in the database named <codeph>default</codeph>. If the tables were in a database other than the default,
+          we would issue a command <codeph>use <varname>db_name</varname> </codeph> to switch to that database
+          before examining or querying its tables. We could also qualify the name of a table by prepending the
+          database name, for example <codeph>default.customer</codeph> and <codeph>default.customer_name</codeph>.
+        </p>
+
+<codeblock>[impala-host:21000] &gt; show databases
+Query finished, fetching results ...
+default
+Returned 1 row(s) in 0.00s
+[impala-host:21000] &gt; show tables
+Query finished, fetching results ...
+customer
+customer_address
+Returned 2 row(s) in 0.00s
+[impala-host:21000] &gt; describe customer_address
++------------------+--------+---------+
+| name             | type   | comment |
++------------------+--------+---------+
+| ca_address_sk    | int    |         |
+| ca_address_id    | string |         |
+| ca_street_number | string |         |
+| ca_street_name   | string |         |
+| ca_street_type   | string |         |
+| ca_suite_number  | string |         |
+| ca_city          | string |         |
+| ca_county        | string |         |
+| ca_state         | string |         |
+| ca_zip           | string |         |
+| ca_country       | string |         |
+| ca_gmt_offset    | float  |         |
+| ca_location_type | string |         |
++------------------+--------+---------+
+Returned 13 row(s) in 0.01
+</codeblock>
+      </conbody>
+    </concept>
+
+    <concept id="tutorial_query_impala">
+
+      <title>Query the Impala Table</title>
+
+      <conbody>
+
+        <p>
+          You can query data contained in the tables. Impala coordinates the query execution across a single node
+          or multiple nodes depending on your configuration, without the overhead of running MapReduce jobs to
+          perform the intermediate processing.
+        </p>
+
+        <p>
+          There are a variety of ways to execute queries on Impala:
+        </p>
+
+        <ul>
+          <li>
+            Using the <codeph>impala-shell</codeph> command in interactive mode:
+<codeblock>$ impala-shell -i impala-host
+Connected to localhost:21000
+[impala-host:21000] &gt; select count(*) from customer_address;
+50000
+Returned 1 row(s) in 0.37s
+</codeblock>
+          </li>
+
+          <li>
+            Passing a set of commands contained in a file:
+<codeblock>$ impala-shell -i impala-host -f myquery.sql
+Connected to localhost:21000
+50000
+Returned 1 row(s) in 0.19s</codeblock>
+          </li>
+
+          <li>
+            Passing a single command to the <codeph>impala-shell</codeph> command. The query is executed, the
+            results are returned, and the shell exits. Make sure to quote the command, preferably with single
+            quotation marks to avoid shell expansion of characters such as <codeph>*</codeph>.
+<codeblock>$ impala-shell -i impala-host -q 'select count(*) from customer_address'
+Connected to localhost:21000
+50000
+Returned 1 row(s) in 0.29s</codeblock>
+          </li>
+        </ul>
+      </conbody>
+    </concept>
+
+    <concept id="tutorial_etl">
+
+      <title>Data Loading and Querying Examples</title>
+
+      <conbody>
+
+        <p>
+          This section describes how to create some sample tables and load data into them. These tables can then be
+          queried using the Impala shell.
+        </p>
+      </conbody>
+
+      <concept id="tutorial_loading">
+
+        <title>Loading Data</title>
+
+        <conbody>
+
+          <p>
+            Loading data involves:
+          </p>
+
+          <ul>
+            <li>
+              Establishing a data set. The example below uses <codeph>.csv</codeph> files.
+            </li>
+
+            <li>
+              Creating tables to which to load data.
+            </li>
+
+            <li>
+              Loading the data into the tables you created.
+            </li>
+          </ul>
+
+<!--
+      <section id="tut_hdfs_perms">
+
+        <title>
+          Modifying Directory Permissions
+        </title>
+
+        <p>
+          Change permission settings so Hive and Impala are both able read and write to the Hive warehouse directory.
+          This process involves changing permissions for the root warehouse directory, then modifying the client
+          <codeph>hive-site.xml</codeph> file so newly created files inherit these permissions.
+        </p>
+
+        <p>
+          Begin by modifying the file permissions. For example, if the warehouse directory is
+          <codeph>/user/hive/warehouse</codeph>, issue the following command:
+        </p>
+
+<codeblock>hdfs dfs -chmod -R 775 /user/hive/warehouse</codeblock>
+        <p>
+          Add the following property to the Hive client <codeph>hive-site.xml</codeph> file:
+        </p>
+
+<codeblock>&lt;property&gt;
+  &lt;name&gt;hive.warehouse.subdir.inherit.perms&lt;/name&gt;
+  &lt;value&gt;true&lt;/value&gt;
+  &lt;description&gt;Set this to true if the table directories should inherit the
+    permission of the warehouse or database directory instead of being created
+    with the permissions derived from dfs umask&lt;/description&gt;
+&lt;/property&gt;</codeblock>
+
+      </section>
+-->
+
+<!-- To do: This is from old days when Impala didn't support LOAD DATA; re-do to show in impala-shell.
+      <section id="tut_loading_csv">
+
+        <title>
+          Loading .csv Data into Tables
+        </title>
+
+        <p>
+          In the Hive shell, load data into TAB1 and TAB2:
+        </p>
+
+
+<codeblock>LOAD DATA LOCAL INPATH 'tab1.csv' OVERWRITE INTO TABLE tab1;
+LOAD DATA LOCAL INPATH 'tab2.csv' OVERWRITE INTO TABLE tab2;</codeblock>
+
+      </section>
+-->
+
+<!--
+      <section id="tut_updating_metadata">
+
+        <title>
+          Updating Metadata
+        </title>
+
+        <p>
+          While you have created new tables and loaded data, Impala does not have information about these new tables. To
+          enable Impala to work with the data in these new tables, connect to an Impala host and then refresh the Impala
+          metadata. For example, if you had an Impala host named <codeph>impala-host</codeph> using the default
+          port, you would issue the following command:
+        </p>
+
+<codeblock>$ impala-shell
+[Not connected] &gt; connect impala-host
+[impala-host:21000] &gt; refresh</codeblock>
+
+      </section>
+-->
+        </conbody>
+      </concept>
+
+      <concept id="tutorial_queries">
+
+        <title>Sample Queries</title>
+
+        <conbody>
+
+          <p>
+            To run these sample queries, create a SQL query file <codeph>query.sql</codeph>, copy and paste each
+            query into the query file, and then run the query file using the shell. For example, to run
+            <codeph>query.sql</codeph> on <codeph>impala-host</codeph>, you might use the command:
+          </p>
+
+<codeblock>impala-shell.sh -i impala-host -f query.sql</codeblock>
+
+          <p>
+            The examples and results below assume you have loaded the sample data into the tables as described
+            above.
+          </p>
+
+          <example>
+
+            <title>Example: Examining Contents of Tables</title>
+
+            <p>
+              Let's start by verifying that the tables do contain the data we expect. Because Impala often deals
+              with tables containing millions or billions of rows, when examining tables of unknown size, include
+              the <codeph>LIMIT</codeph> clause to avoid huge amounts of unnecessary output, as in the final query.
+              (If your interactive query starts displaying an unexpected volume of data, press
+              <codeph>Ctrl-C</codeph> in <codeph>impala-shell</codeph> to cancel the query.)
+            </p>
+
+<codeblock>SELECT * FROM tab1;
+SELECT * FROM tab2;
+SELECT * FROM tab2 LIMIT 5;</codeblock>
+
+            <p>
+              Results:
+            </p>
+
+<codeblock>+----+-------+------------+-------------------------------+
+| id | col_1 | col_2      | col_3                         |
++----+-------+------------+-------------------------------+
+| 1  | true  | 123.123    | 2012-10-24 08:55:00           |
+| 2  | false | 1243.5     | 2012-10-25 13:40:00           |
+| 3  | false | 24453.325  | 2008-08-22 09:33:21.123000000 |
+| 4  | false | 243423.325 | 2007-05-12 22:32:21.334540000 |
+| 5  | true  | 243.325    | 1953-04-22 09:11:33           |
++----+-------+------------+-------------------------------+
+
++----+-------+---------------+
+| id | col_1 | col_2         |
++----+-------+---------------+
+| 1  | true  | 12789.123     |
+| 2  | false | 1243.5        |
+| 3  | false | 24453.325     |
+| 4  | false | 2423.3254     |
+| 5  | true  | 243.325       |
+| 60 | false | 243565423.325 |
+| 70 | true  | 243.325       |
+| 80 | false | 243423.325    |
+| 90 | true  | 243.325       |
++----+-------+---------------+
+
++----+-------+-----------+
+| id | col_1 | col_2     |
++----+-------+-----------+
+| 1  | true  | 12789.123 |
+| 2  | false | 1243.5    |
+| 3  | false | 24453.325 |
+| 4  | false | 2423.3254 |
+| 5  | true  | 243.325   |
++----+-------+-----------+</codeblock>
+
+          </example>
+
+          <example>
+
+            <title>Example: Aggregate and Join</title>
+
+<codeblock>SELECT tab1.col_1, MAX(tab2.col_2), MIN(tab2.col_2)
+FROM tab2 JOIN tab1 USING (id)
+GROUP BY col_1 ORDER BY 1 LIMIT 5;</codeblock>
+
+            <p>
+              Results:
+            </p>
+
+<codeblock>+-------+-----------------+-----------------+
+| col_1 | max(tab2.col_2) | min(tab2.col_2) |
++-------+-----------------+-----------------+
+| false | 24453.325       | 1243.5          |
+| true  | 12789.123       | 243.325         |
++-------+-----------------+-----------------+</codeblock>
+
+          </example>
+
+          <example>
+
+            <title>Example: Subquery, Aggregate and Joins</title>
+
+<codeblock>SELECT tab2.*
+FROM tab2,
+(SELECT tab1.col_1, MAX(tab2.col_2) AS max_col2
+ FROM tab2, tab1
+ WHERE tab1.id = tab2.id
+ GROUP BY col_1) subquery1
+WHERE subquery1.max_col2 = tab2.col_2;</codeblock>
+
+            <p>
+              Results:
+            </p>
+
+<codeblock>+----+-------+-----------+
+| id | col_1 | col_2     |
++----+-------+-----------+
+| 1  | true  | 12789.123 |
+| 3  | false | 24453.325 |
++----+-------+-----------+</codeblock>
+
+          </example>
+
+          <example>
+
+            <title>Example: INSERT Query</title>
+
+<codeblock>INSERT OVERWRITE TABLE tab3
+SELECT id, col_1, col_2, MONTH(col_3), DAYOFMONTH(col_3)
+FROM tab1 WHERE YEAR(col_3) = 2012;</codeblock>
+
+            <p>
+              Query <codeph>TAB3</codeph> to check the result:
+            </p>
+
+<codeblock>SELECT * FROM tab3;
+</codeblock>
+
+            <p>
+              Results:
+            </p>
+
+<codeblock>+----+-------+---------+-------+-----+
+| id | col_1 | col_2   | month | day |
++----+-------+---------+-------+-----+
+| 1  | true  | 123.123 | 10    | 24  |
+| 2  | false | 1243.5  | 10    | 25  |
++----+-------+---------+-------+-----+</codeblock>
+
+          </example>
+        </conbody>
+      </concept>
+    </concept>
+  </concept>
+
+  <concept id="tut_advanced">
+
+    <title>Advanced Tutorials</title>
+
+    <conbody>
+
+      <p>
+        These tutorials walk you through advanced scenarios or specialized features.
+      </p>
+
+      <p outputclass="toc inpage"/>
+    </conbody>
+
+    <concept id="tut_external_partition_data">
+
+      <title>Attaching an External Partitioned Table to an HDFS Directory Structure</title>
+
+      <conbody>
+
+        <p>
+          This tutorial shows how you might set up a directory tree in HDFS, put data files into the lowest-level
+          subdirectories, and then use an Impala external table to query the data files from their original
+          locations.
+        </p>
+
+        <p>
+          The tutorial uses a table with web log data, with separate subdirectories for the year, month, day, and
+          host. For simplicity, we use a tiny amount of CSV data, loading the same data into each partition.
+        </p>
+
+        <p>
+          First, we make an Impala partitioned table for CSV data, and look at the underlying HDFS directory
+          structure to understand the directory structure to re-create elsewhere in HDFS. The columns
+          <codeph>field1</codeph>, <codeph>field2</codeph>, and <codeph>field3</codeph> correspond to the contents
+          of the CSV data files. The <codeph>year</codeph>, <codeph>month</codeph>, <codeph>day</codeph>, and
+          <codeph>host</codeph> columns are all represented as subdirectories within the table structure, and are
+          not part of the CSV files. We use <codeph>STRING</codeph> for each of these columns so that we can
+          produce consistent subdirectory names, with leading zeros for a consistent length.
+        </p>
+
+<codeblock>create database external_partitions;
+use external_partitions;
+create table logs (field1 string, field2 string, field3 string)
+  partitioned by (year string, month string , day string, host string)
+  row format delimited fields terminated by ',';
+insert into logs partition (year="2013", month="07", day="28", host="host1") values ("foo","foo","foo");
+insert into logs partition (year="2013", month="07", day="28", host="host2") values ("foo","foo","foo");
+insert into logs partition (year="2013", month="07", day="29", host="host1") values ("foo","foo","foo");
+insert into logs partition (year="2013", month="07", day="29", host="host2") values ("foo","foo","foo");
+insert into logs partition (year="2013", month="08", day="01", host="host1") values ("foo","foo","foo");
+</codeblock>
+
+        <p>
+          Back in the Linux shell, we examine the HDFS directory structure. (Your Impala data directory might be in
+          a different location; for historical reasons, it is sometimes under the HDFS path
+          <filepath>/user/hive/warehouse</filepath>.) We use the <codeph>hdfs dfs -ls</codeph> command to examine
+          the nested subdirectories corresponding to each partitioning column, with separate subdirectories at each
+          level (with <codeph>=</codeph> in their names) representing the different values for each partitioning
+          column. When we get to the lowest level of subdirectory, we use the <codeph>hdfs dfs -cat</codeph>
+          command to examine the data file and see CSV-formatted data produced by the <codeph>INSERT</codeph>
+          statement in Impala.
+        </p>
+
+<codeblock>$ hdfs dfs -ls /user/impala/warehouse/external_partitions.db
+Found 1 items
+drwxrwxrwt   - impala hive          0 2013-08-07 12:24 /user/impala/warehouse/external_partitions.db/logs
+$ hdfs dfs -ls /user/impala/warehouse/external_partitions.db/logs
+Found 1 items
+drwxr-xr-x   - impala hive          0 2013-08-07 12:24 /user/impala/warehouse/external_partitions.db/logs/year=2013
+$ hdfs dfs -ls /user/impala/warehouse/external_partitions.db/logs/year=2013
+Found 2 items
+drwxr-xr-x   - impala hive          0 2013-08-07 12:23 /user/impala/warehouse/external_partitions.db/logs/year=2013/month=07
+drwxr-xr-x   - impala hive          0 2013-08-07 12:24 /user/impala/warehouse/external_partitions.db/logs/year=2013/month=08
+$ hdfs dfs -ls /user/impala/warehouse/external_partitions.db/logs/year=2013/month=07
+Found 2 items
+drwxr-xr-x   - impala hive          0 2013-08-07 12:22 /user/impala/warehouse/external_partitions.db/logs/year=2013/month=07/day=28
+drwxr-xr-x   - impala hive          0 2013-08-07 12:23 /user/impala/warehouse/external_partitions.db/logs/year=2013/month=07/day=29
+$ hdfs dfs -ls /user/impala/warehouse/external_partitions.db/logs/year=2013/month=07/day=28
+Found 2 items
+drwxr-xr-x   - impala hive          0 2013-08-07 12:21 /user/impala/warehouse/external_partitions.db/logs/year=2013/month=07/day=28/host=host1
+drwxr-xr-x   - impala hive          0 2013-08-07 12:22 /user/impala/warehouse/external_partitions.db/logs/year=2013/month=07/day=28/host=host2
+$ hdfs dfs -ls /user/impala/warehouse/external_partitions.db/logs/year=2013/month=07/day=28/host=host1
+Found 1 items
+-rw-r--r--   3 impala hive         12 2013-08-07 12:21 /user/impala/warehouse/external_partiti
+ons.db/logs/year=2013/month=07/day=28/host=host1/3981726974111751120--8907184999369517436_822630111_data.0
+$ hdfs dfs -cat /user/impala/warehouse/external_partitions.db/logs/year=2013/month=07/day=28/\
+host=host1/3981726974111751120--8 907184999369517436_822630111_data.0
+foo,foo,foo
+</codeblock>
+
+        <p>
+          Still in the Linux shell, we use <codeph>hdfs dfs -mkdir</codeph> to create several data directories
+          outside the HDFS directory tree that Impala controls (<filepath>/user/impala/warehouse</filepath> in this
+          example, maybe different in your case). Depending on your configuration, you might need to log in as a
+          user with permission to write into this HDFS directory tree; for example, the commands shown here were
+          run while logged in as the <codeph>hdfs</codeph> user.
+        </p>
+
+<codeblock>$ hdfs dfs -mkdir -p /user/impala/data/logs/year=2013/month=07/day=28/host=host1
+$ hdfs dfs -mkdir -p /user/impala/data/logs/year=2013/month=07/day=28/host=host2
+$ hdfs dfs -mkdir -p /user/impala/data/logs/year=2013/month=07/day=28/host=host1
+$ hdfs dfs -mkdir -p /user/impala/data/logs/year=2013/month=07/day=29/host=host1
+$ hdfs dfs -mkdir -p /user/impala/data/logs/year=2013/month=08/day=01/host=host1
+</codeblock>
+
+        <p>
+          We make a tiny CSV file, with values different than in the <codeph>INSERT</codeph> statements used
+          earlier, and put a copy within each subdirectory that we will use as an Impala partition.
+        </p>
+
+<codeblock>$ cat &gt;dummy_log_data
+bar,baz,bletch
+$ hdfs dfs -mkdir -p /user/impala/data/external_partitions/year=2013/month=08/day=01/host=host1
+$ hdfs dfs -mkdir -p /user/impala/data/external_partitions/year=2013/month=07/day=28/host=host1
+$ hdfs dfs -mkdir -p /user/impala/data/external_partitions/year=2013/month=07/day=28/host=host2
+$ hdfs dfs -mkdir -p /user/impala/data/external_partitions/year=2013/month=07/day=29/host=host1
+$ hdfs dfs -put dummy_log_data /user/impala/data/logs/year=2013/month=07/day=28/host=host1
+$ hdfs dfs -put dummy_log_data /user/impala/data/logs/year=2013/month=07/day=28/host=host2
+$ hdfs dfs -put dummy_log_data /user/impala/data/logs/year=2013/month=07/day=29/host=host1
+$ hdfs dfs -put dummy_log_data /user/impala/data/logs/year=2013/month=08/day=01/host=host1
+</codeblock>
+
+        <p>
+          Back in the <cmdname>impala-shell</cmdname> interpreter, we move the original Impala-managed table aside,
+          and create a new <i>external</i> table with a <codeph>LOCATION</codeph> clause pointing to the directory
+          under which we have set up all the partition subdirectories and data files.
+        </p>
+
+<codeblock>use external_partitions;
+alter table logs rename to logs_original;
+create external table logs (field1 string, field2 string, field3 string)
+  partitioned by (year string, month string, day string, host string)
+  row format delimited fields terminated by ','
+  location '/user/impala/data/logs';
+</codeblock>
+
+        <p>
+          Because partition subdirectories and data files come and go during the data lifecycle, you must identify
+          each of the partitions through an <codeph>ALTER TABLE</codeph> statement before Impala recognizes the
+          data files they contain.
+        </p>
+
+<codeblock>alter table logs add partition (year="2013",month="07",day="28",host="host1")
+alter table log_type add partition (year="2013",month="07",day="28",host="host2");
+alter table log_type add partition (year="2013",month="07",day="29",host="host1");
+alter table log_type add partition (year="2013",month="08",day="01",host="host1");
+</codeblock>
+
+        <p>
+          We issue a <codeph>REFRESH</codeph> statement for the table, always a safe practice when data files have
+          been manually added, removed, or changed. Then the data is ready to be queried. The <codeph>SELECT
+          *</codeph> statement illustrates that the data from our trivial CSV file was recognized in each of the
+          partitions where we copied it. Although in this case there are only a few rows, we include a
+          <codeph>LIMIT</codeph> clause on this test query just in case there is more data than we expect.
+        </p>
+
+<codeblock>refresh log_type;
+select * from log_type limit 100;
++--------+--------+--------+------+-------+-----+-------+
+| field1 | field2 | field3 | year | month | day | host  |
++--------+--------+--------+------+-------+-----+-------+
+| bar    | baz    | bletch | 2013 | 07    | 28  | host1 |
+| bar    | baz    | bletch | 2013 | 08    | 01  | host1 |
+| bar    | baz    | bletch | 2013 | 07    | 29  | host1 |
+| bar    | baz    | bletch | 2013 | 07    | 28  | host2 |
++--------+--------+--------+------+-------+-----+-------+
+</codeblock>
+      </conbody>
+    </concept>
+
+    <concept id="tutorial_impala_hive">
+
+      <title>Switching Back and Forth Between Impala and Hive</title>
+
+      <conbody>
+
+        <p>
+          Sometimes, you might find it convenient to switch to the Hive shell to perform some data loading or
+          transformation operation, particularly on file formats such as RCFile, SequenceFile, and Avro that Impala
+          currently can query but not write to.
+        </p>
+
+        <p>
+          Whenever you create, drop, or alter a table or other kind of object through Hive, the next time you
+          switch back to the <cmdname>impala-shell</cmdname> interpreter, issue a one-time <codeph>INVALIDATE
+          METADATA</codeph> statement so that Impala recognizes the new or changed object.
+        </p>
+
+        <p>
+          Whenever you load, insert, or change data in an existing table through Hive (or even through manual HDFS
+          operations such as the <cmdname>hdfs</cmdname> command), the next time you switch back to the
+          <cmdname>impala-shell</cmdname> interpreter, issue a one-time <codeph>REFRESH
+          <varname>table_name</varname></codeph> statement so that Impala recognizes the new or changed data.
+        </p>
+
+        <p>
+          For examples showing how this process works for the <codeph>REFRESH</codeph> statement, look at the
+          examples of creating RCFile and SequenceFile tables in Impala, loading data through Hive, and then
+          querying the data through Impala. See <xref href="impala_rcfile.xml#rcfile"/> and
+          <xref href="impala_seqfile.xml#seqfile"/> for those examples.
+        </p>
+
+        <p>
+          For examples showing how this process works for the <codeph>INVALIDATE METADATA</codeph> statement, look
+          at the example of creating and loading an Avro table in Hive, and then querying the data through Impala.
+          See <xref href="impala_avro.xml#avro"/> for that example.
+        </p>
+
+        <note rev="1.2.0">
+          <p rev="1.2.0">
+            Originally, Impala did not support UDFs, but this feature is available in Impala starting in Impala
+            1.2. Some <codeph>INSERT ... SELECT</codeph> transformations that you originally did through Hive can
+            now be done through Impala. See <xref href="impala_udf.xml#udfs"/> for details.
+          </p>
+
+          <p rev="1.2.0">
+            Prior to Impala 1.2, the <codeph>REFRESH</codeph> and <codeph>INVALIDATE METADATA</codeph> statements
+            needed to be issued on each Impala node to which you connected and issued queries. In Impala 1.2 and
+            higher, when you issue either of those statements on any Impala node, the results are broadcast to all
+            the Impala nodes in the cluster, making it truly a one-step operation after each round of DDL or ETL
+            operations in Hive.
+          </p>
+        </note>
+      </conbody>
+    </concept>
+
+    <concept rev="1.2.2" id="tut_cross_join">
+
+      <title>Cross Joins and Cartesian Products with the CROSS JOIN Operator</title>
+
+      <conbody>
+
+        <p>
+          Originally, Impala restricted join queries so that they had to include at least one equality comparison
+          between the columns of the tables on each side of the join operator. With the huge tables typically
+          processed by Impala, any miscoded query that produced a full Cartesian product as a result set could
+          consume a huge amount of cluster resources.
+        </p>
+
+        <p>
+          In Impala 1.2.2 and higher, this restriction is lifted when you use the <codeph>CROSS JOIN</codeph>
+          operator in the query. You still cannot remove all <codeph>WHERE</codeph> clauses from a query like
+          <codeph>SELECT * FROM t1 JOIN t2</codeph> to produce all combinations of rows from both tables. But you
+          can use the <codeph>CROSS JOIN</codeph> operator to explicitly request such a Cartesian product.
+          Typically, this operation is applicable for smaller tables, where the result set still fits within the
+          memory of a single Impala node.
+        </p>
+
+        <p>
+          The following example sets up data for use in a series of comic books where characters battle each other.
+          At first, we use an equijoin query, which only allows characters from the same time period and the same
+          planet to meet.
+        </p>
+
+<codeblock>[localhost:21000] &gt; create table heroes (name string, era string, planet string);
+[localhost:21000] &gt; create table villains (name string, era string, planet string);
+[localhost:21000] &gt; insert into heroes values
+                  &gt; ('Tesla','20th century','Earth'),
+                  &gt; ('Pythagoras','Antiquity','Earth'),
+                  &gt; ('Zopzar','Far Future','Mars');
+Inserted 3 rows in 2.28s
+[localhost:21000] &gt; insert into villains values
+                  &gt; ('Caligula','Antiquity','Earth'),
+                  &gt; ('John Dillinger','20th century','Earth'),
+                  &gt; ('Xibulor','Far Future','Venus');
+Inserted 3 rows in 1.93s
+[localhost:21000] &gt; select concat(heroes.name,' vs. ',villains.name) as battle
+                  &gt; from heroes join villains
+                  &gt; where heroes.era = villains.era and heroes.planet = villains.planet;
++--------------------------+
+| battle                   |
++--------------------------+
+| Tesla vs. John Dillinger |
+| Pythagoras vs. Caligula  |
++--------------------------+
+Returned 2 row(s) in 0.47s</codeblock>
+
+        <p>
+          Readers demanded more action, so we added elements of time travel and space travel so that any hero could
+          face any villain. Prior to Impala 1.2.2, this type of query was impossible because all joins had to
+          reference matching values between the two tables:
+        </p>
+
+<codeblock>[localhost:21000] &gt; -- Cartesian product not possible in Impala 1.1.
+                  &gt; select concat(heroes.name,' vs. ',villains.name) as battle from heroes join villains;
+ERROR: NotImplementedException: Join between 'heroes' and 'villains' requires at least one conjunctive equality predicate between the two tables</codeblock>
+
+        <p>
+          With Impala 1.2.2, we rewrite the query slightly to use <codeph>CROSS JOIN</codeph> rather than
+          <codeph>JOIN</codeph>, and now the result set includes all combinations:
+        </p>
+
+<codeblock>[localhost:21000] &gt; -- Cartesian product available in Impala 1.2.2 with the CROSS JOIN syntax.
+                  &gt; select concat(heroes.name,' vs. ',villains.name) as battle from heroes cross join villains;
++-------------------------------+
+| battle                        |
++-------------------------------+
+| Tesla vs. Caligula            |
+| Tesla vs. John Dillinger      |
+| Tesla vs. Xibulor             |
+| Pythagoras vs. Caligula       |
+| Pythagoras vs. John Dillinger |
+| Pythagoras vs. Xibulor        |
+| Zopzar vs. Caligula           |
+| Zopzar vs. John Dillinger     |
+| Zopzar vs. Xibulor            |
++-------------------------------+
+Returned 9 row(s) in 0.33s</codeblock>
+
+        <p>
+          The full combination of rows from both tables is known as the Cartesian product. This type of result set
+          is often used for creating grid data structures. You can also filter the result set by including
+          <codeph>WHERE</codeph> clauses that do not explicitly compare columns between the two tables. The
+          following example shows how you might produce a list of combinations of year and quarter for use in a
+          chart, and then a shorter list with only selected quarters.
+        </p>
+
+<codeblock>[localhost:21000] &gt; create table x_axis (x int);
+[localhost:21000] &gt; create table y_axis (y int);
+[localhost:21000] &gt; insert into x_axis values (1),(2),(3),(4);
+Inserted 4 rows in 2.14s
+[localhost:21000] &gt; insert into y_axis values (2010),(2011),(2012),(2013),(2014);
+Inserted 5 rows in 1.32s
+[localhost:21000] &gt; select y as year, x as quarter from x_axis cross join y_axis;
++------+---------+
+| year | quarter |
++------+---------+
+| 2010 | 1       |
+| 2011 | 1       |
+| 2012 | 1       |
+| 2013 | 1       |
+| 2014 | 1       |
+| 2010 | 2       |
+| 2011 | 2       |
+| 2012 | 2       |
+| 2013 | 2       |
+| 2014 | 2       |
+| 2010 | 3       |
+| 2011 | 3       |
+| 2012 | 3       |
+| 2013 | 3       |
+| 2014 | 3       |
+| 2010 | 4       |
+| 2011 | 4       |
+| 2012 | 4       |
+| 2013 | 4       |
+| 2014 | 4       |
++------+---------+
+Returned 20 row(s) in 0.38s
+[localhost:21000] &gt; select y as year, x as quarter from x_axis cross join y_axis where x in (1,3);
++------+---------+
+| year | quarter |
++------+---------+
+| 2010 | 1       |
+| 2011 | 1       |
+| 2012 | 1       |
+| 2013 | 1       |
+| 2014 | 1       |
+| 2010 | 3       |
+| 2011 | 3       |
+| 2012 | 3       |
+| 2013 | 3       |
+| 2014 | 3       |
++------+---------+
+Returned 10 row(s) in 0.39s</codeblock>
+      </conbody>
+    </concept>
+
+    <concept audience="Cloudera" id="tut_connect">
+
+      <title>Connecting to Impala through impala-shell</title>
+
+      <conbody>
+
+        <p>
+          This tutorial provides some tips for connecting to Impala through the <cmdname>impala-shell</cmdname>
+          interpreter, and gives usage tips for particular scenarios.
+        </p>
+
+        <ul>
+          <li>
+            <p>
+              By default, <cmdname>impala-shell</cmdname> with no arguments connects to the current host, port
+              21000.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              <codeph>-i localhost</codeph> option gives you a shorter command prompt.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              The <codeph>--quiet</codeph> option suppresses some informational messages, such as the
+              <codeph>Query:</codeph> line that echoes each command entered.
+            </p>
+          </li>
+
+          <li>
+            <codeph>CONNECT <varname>hostname</varname></codeph> lets you switch to another host (keeping the same
+            default port 21000).
+          </li>
+
+          <li>
+            <p>
+              You could run <cmdname>impala-shell</cmdname> from a completely different host to keep login accounts
+              off the actual Impala nodes.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              For load balancing, you would connect to different coordinator nodes for concurrent queries.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              For benchmarking, you would skip pretty printing with the <codeph>-B</codeph> option, or even skip
+              all screen output with the <codeph>-o</codeph> option or shell redirection.
+            </p>
+          </li>
+        </ul>
+      </conbody>
+    </concept>
+  </concept>
+
+  <concept audience="Cloudera" id="tut_mem_limit">
+
+    <title>Effects of Memory Limits on Impala Queries</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Memory"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p></p>
+    </conbody>
+  </concept>
+
+  <concept id="tut_parquet_schemaless">
+
+    <title>Dealing with Parquet Files with Unknown Schema</title>
+    <prolog>
+      <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Tutorials"/>
+      <data name="Category" value="Schemas"/>
+      <data name="Category" value="Parquet"/>
+      <data name="Category" value="File Formats"/>
+      </metadata>
+    </prolog>
+
+    <conbody>
+
+      <p>
+        As data pipelines start to include more aspects such as NoSQL or loosely specified schemas, you might encounter
+        situations where you have data files (particularly in Parquet format) where you do not know the precise table definition.
+        This tutorial shows how you can build an Impala table around data that comes from non-Impala or even non-SQL sources,
+        where you do not have control of the table layout and might not be familiar with the characteristics of the data.
+      </p>
+
+<p>
+The data used in this tutorial represents airline on-time arrival statistics, from October 1987 through April 2008.
+See the details on the <xref href="http://stat-computing.org/dataexpo/2009/" scope="external" format="html">2009 ASA Data Expo web site</xref>.
+You can also see the <xref href="http://stat-computing.org/dataexpo/2009/the-data.html" scope="external" format="html">explanations of the columns</xref>;
+for purposes of this exercise, wait until after following the tutorial before examining the schema, to better simulate
+a real-life situation where you cannot rely on assumptions and assertions about the ranges and representations of
+data values.
+</p>
+
+<p>
+We will download Parquet files containing this data from the Ibis blog.
+First, we download and unpack the data files.
+There are 8 files totalling 1.4 GB.
+Each file is less than 256 MB.
+</p>
+
+<codeblock>$ wget -O airlines_parquet.tar.gz https://www.dropbox.com/s/ol9x51tqp6cv4yc/airlines_parquet.tar.gz?dl=0
+...
+Length: 1245204740 (1.2G) [application/octet-stream]
+Saving to: \u201cairlines_parquet.tar.gz\u201d
+
+2015-08-12 17:14:24 (23.6 MB/s) - \u201cairlines_parquet.tar.gz\u201d saved [1245204740/1245204740]
+
+$ tar xvzf airlines_parquet.tar.gz
+airlines_parquet/
+airlines_parquet/93459d994898a9ba-77674173b331fa9a_2073981944_data.0.parq
+airlines_parquet/93459d994898a9ba-77674173b331fa99_1555718317_data.1.parq
+airlines_parquet/93459d994898a9ba-77674173b331fa99_1555718317_data.0.parq
+airlines_parquet/93459d994898a9ba-77674173b331fa96_2118228804_data.0.parq
+airlines_parquet/93459d994898a9ba-77674173b331fa97_574780876_data.0.parq
+airlines_parquet/93459d994898a9ba-77674173b331fa96_2118228804_data.1.parq
+airlines_parquet/93459d994898a9ba-77674173b331fa98_1194408366_data.0.parq
+airlines_parquet/93459d994898a9ba-77674173b331fa9b_1413430552_data.0.parq
+$ cd airlines_parquet/
+$ du -kch *.parq
+253M  93459d994898a9ba-77674173b331fa96_2118228804_data.0.parq
+65M 93459d994898a9ba-77674173b331fa96_2118228804_data.1.parq
+156M  93459d994898a9ba-77674173b331fa97_574780876_data.0.parq
+240M  93459d994898a9ba-77674173b331fa98_1194408366_data.0.parq
+253M  93459d994898a9ba-77674173b331fa99_1555718317_data.0.parq
+16M 93459d994898a9ba-77674173b331fa99_1555718317_data.1.parq
+177M  93459d994898a9ba-77674173b331fa9a_2073981944_data.0.parq
+213M  93459d994898a9ba-77674173b331fa9b_1413430552_data.0.parq
+1.4G  total
+</codeblock>
+
+<p>
+Next, we put the Parquet data files in HDFS, all together in a single directory,
+with permissions on the directory and the files so that the <codeph>impala</codeph>
+user will be able to read them.
+</p>
+
+<note>
+After unpacking, we saw the largest Parquet file was 253 MB.
+When copying Parquet files into HDFS for Impala to use,
+for maximum query performance, make sure that each file resides in a single HDFS data block.
+Therefore, we pick a size larger than any single file and specify that as the block size, using the argument
+<codeph>-Ddfs.block.size=256m</codeph> on the <codeph>hdfs dfs -put</codeph> command.
+</note>
+
+<codeblock>$ hdfs dfs -mkdir -p hdfs://demo_host.example.com:8020/user/impala/staging/airlines
+$ hdfs dfs -Ddfs.block.size=256m -put *.parq /user/impala/staging/airlines
+$ hdfs dfs -ls /user/impala/staging
+Found 1 items
+drwxrwxrwx   - hdfs supergroup          0 2015-08-12 13:52 /user/impala/staging/airlines
+$ hdfs dfs -ls hdfs://demo_host.example.com:8020/user/impala/staging/airlines
+Found 8 items
+-rw-r--r--   3 jrussell supergroup  265107489 2015-08-12 17:18 /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa96_2118228804_data.0.parq
+-rw-r--r--   3 jrussell supergroup   67544715 2015-08-12 17:18 /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa96_2118228804_data.1.parq
+-rw-r--r--   3 jrussell supergroup  162556490 2015-08-12 17:18 /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa97_574780876_data.0.parq
+-rw-r--r--   3 jrussell supergroup  251603518 2015-08-12 17:18 /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa98_1194408366_data.0.parq
+-rw-r--r--   3 jrussell supergroup  265186603 2015-08-12 17:18 /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa99_1555718317_data.0.parq
+-rw-r--r--   3 jrussell supergroup   16663754 2015-08-12 17:18 /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa99_1555718317_data.1.parq
+-rw-r--r--   3 jrussell supergroup  185511677 2015-08-12 17:18 /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa9a_2073981944_data.0.parq
+-rw-r--r--   3 jrussell supergroup  222794621 2015-08-12 17:18 /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa9b_1413430552_data.0.parq
+</codeblock>
+
+<p>
+With the files in an accessible location in HDFS, we create a database table that uses the data in those files.
+The <codeph>CREATE EXTERNAL</codeph> syntax and the <codeph>LOCATION</codeph> attribute point Impala at the appropriate HDFS directory.
+The <codeph>LIKE PARQUET '<varname>path_to_any_parquet_file</varname>'</codeph> clause means we skip the list of column names and types;
+Impala automatically gets the column names and data types straight from the data files.
+(Currently, this technique only works for Parquet files.)
+We ignore the warning about lack of <codeph>READ_WRITE</codeph> access to the files in HDFS;
+the <codeph>impala</codeph> user can read the files, which will be sufficient for us to experiment with
+queries and perform some copy and transform operations into other tables.
+</p>
+
+<codeblock>$ impala-shell -i localhost
+Starting Impala Shell without Kerberos authentication
+Connected to localhost:21000
+Server version: impalad version 2.2.0-cdh5 RELEASE (build 2ffd73a4255cefd521362ffe1cfb37463f67f75c)
+Welcome to the Impala shell. Press TAB twice to see a list of available commands.
+
+Copyright (c) 2012 Cloudera, Inc. All rights reserved.
+
+(Shell build version: Impala Shell v2.1.2-cdh5 (92438b7) built on Tue Feb 24 12:36:33 PST 2015)
+[localhost:21000] > create database airline_data;
+[localhost:21000] > use airline_data;
+[localhost:21000] > create external table airlines_external
+                  > like parquet 'hdfs://demo_host.example.com:8020/user/impala/staging/airlines/93459d994898a9ba-77674173b331fa96_2118228804_data.0.parq'
+                  > stored as parquet location 'hdfs://demo_host.example.com:8020/user/impala/staging/airlines';
+WARNINGS: Impala does not have READ_WRITE access to path 'hdfs://demo_host.example.com:8020/user/impala/staging'
+</codeblock>
+
+<p>
+With the table created, we examine its physical and logical characteristics to confirm that the data is really
+there and in a format and shape that we can work with.
+The <codeph>SHOW TABLE STATS</codeph> statement gives a very high-level summary of the table,
+showing how many files and how much total data it contains.
+Also, it confirms that the table is expecting all the associated data files to be in Parquet format.
+(The ability to work with all kinds of HDFS data files in different formats means that it is
+possible to have a mismatch between the format of the data files, and the format
+that the table expects the data files to be in.)
+The <codeph>SHOW FILES</codeph> statement confirms that the data in the table has the expected number,
+names, and sizes of the original Parquet files.
+The <codeph>DESCRIBE</codeph> statement (or its abbreviation <codeph>DESC</codeph>) confirms the names and types
+of the columns that Impala automatically created after reading that metadata from the Parquet file.
+The <codeph>DESCRIBE FORMATTED</codeph> statement prints out some extra detail along with the column definitions;
+the pieces we care about for this exercise are the containing database for the table,
+the location of the associated data files in HDFS, the fact that it's an external table so Impala will not
+delete the HDFS files when we finish the experiments and drop the table, and the fact that the
+table is set up to work exclusively with files in the Parquet format.
+</p>
+
+<codeblock>[localhost:21000] > show table stats airlines_external;
++-------+--------+--------+--------------+-------------------+---------+-------------------+
+| #Rows | #Files | Size   | Bytes Cached | Cache Replication | Format  | Incremental stats |
++-------+--------+--------+--------------+-------------------+---------+-------------------+
+| -1    | 8      | 1.34GB | NOT CACHED   | NOT CACHED        | PARQUET | false             |
++-------+--------+--------+--------------+-------------------+---------+-------------------+
+[localhost:21000] > show files in airlines_external;
++----------------------------------------------------------------------------------------+----------+-----------+
+| path                                                                                   | size     | partition |
++----------------------------------------------------------------------------------------+----------+-----------+
+| /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa96_2118228804_data.0.parq | 252.83MB |           |
+| /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa96_2118228804_data.1.parq | 64.42MB  |           |
+| /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa97_574780876_data.0.parq  | 155.03MB |           |
+| /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa98_1194408366_data.0.parq | 239.95MB |           |
+| /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa99_1555718317_data.0.parq | 252.90MB |           |
+| /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa99_1555718317_data.1.parq | 15.89MB  |           |
+| /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa9a_2073981944_data.0.parq | 176.92MB |           |
+| /user/impala/staging/airlines/93459d994898a9ba-77674173b331fa9b_1413430552_data.0.parq | 212.47MB |           |
++----------------------------------------------------------------------------------------+----------+-----------+
+[localhost:21000] > describe airlines_external;
++---------------------+--------+---------------------------------------------------+
+| name                | type   | comment                                           |
++---------------------+--------+---------------------------------------------------+
+| year                | int    | inferred from: optional int32 year                |
+| month               | int    | inferred from: optional int32 month               |
+| day                 | int    | inferred from: optional int32 day                 |
+| dayofweek           | int    | inferred from: optional int32 dayofweek           |
+| dep_time            | int    | inferred from: optional int32 dep_time            |
+| crs_dep_time        | int    | inferred from: optional int32 crs_dep_time        |
+| arr_time            | int    | inferred from: optional int32 arr_time            |
+| crs_arr_time        | int    | inferred from: optional int32 crs_arr_time        |
+| carrier             | string | inferred from: optional binary carrier            |
+| flight_num          | int    | inferred from: optional int32 flight_num          |
+| tail_num            | int    | inferred from: optional int32 tail_num            |
+| actual_elapsed_time | int    | inferred from: optional int32 actual_elapsed_time |
+| crs_elapsed_time    | int    | inferred from: optional int32 crs_elapsed_time    |
+| airtime             | int    | inferred from: optional int32 airtime             |
+| arrdelay            | int    | inferred from: optional int32 arrdelay            |
+| depdelay            | int    | inferred from: optional int32 depdelay            |
+| origin              | string | inferred from: optional binary origin             |
+| dest                | string | inferred from: optional binary dest               |
+| distance            | int    | inferred from: optional int32 distance            |
+| taxi_in             | int    | inferred from: optional int32 taxi_in             |
+| taxi_out            | int    | inferred from: optional int32 taxi_out            |
+| cancelled           | int    | inferred from: optional int32 cancelled           |
+| cancellation_code   | string | inferred from: optional binary cancellation_code  |
+| diverted            | int    | inferred from: optional int32 diverted            |
+| carrier_delay       | int    | inferred from: optional int32 carrier_delay       |
+| weather_delay       | int    | inferred from: optional int32 weather_delay       |
+| nas_delay           | int    | inferred from: optional int32 nas_delay           |
+| security_delay      | int    | inferred from: optional int32 security_delay      |
+| late_aircraft_delay | int    | inferred from: optional int32 late_aircraft_delay |
++---------------------+--------+---------------------------------------------------+
+[localhost:21000] > desc formatted airlines_external;
++------------------------------+-------------------------------
+| name                         | type
++------------------------------+-------------------------------
+...
+| # Detailed Table Information | NULL
+| Database:                    | airline_data
+| Owner:                       | jrussell
+...
+| Location:                    | /user/impala/staging/airlines
+| Table Type:                  | EXTERNAL_TABLE
+...
+| # Storage Information        | NULL
+| SerDe Library:               | parquet.hive.serde.ParquetHiveSerDe
+| InputFormat:                 | parquet.hive.DeprecatedParquetInputFormat
+| OutputFormat:                | parquet.hive.DeprecatedParquetOutputFormat
+...
+</codeblock>
+
+<p>
+Now that we are confident that the connections are solid between the Impala table and the
+underlying Parquet files, we run some initial queries to understand the characteristics
+of the data: the overall number of rows, and the ranges and how many
+different values are in certain columns.
+For convenience in understanding the magnitude of the <codeph>COUNT(*)</codeph>
+result, we run another query dividing the number of rows by 1 million, demonstrating that there are 123 million rows in the table.
+</p>
+
+<!-- I think this is the very longest line that could possibly fit without wrapping in the PDF output; 87 characters wide.
+[localhost:21000] > select count(*) / 1e6 as 'millions of rows' from airlines_external;
+-->
+
+<codeblock>[localhost:21000] > select count(*) from airlines_external;
++-----------+
+| count(*)  |
++-----------+
+| 123534969 |
++-----------+
+Fetched 1 row(s) in 1.32s
+[localhost:21000] > select count(*) / 1e6 as 'millions of rows' from airlines_external;
++------------------+
+| millions of rows |
++------------------+
+| 123.534969       |
++------------------+
+Fetched 1 row(s) in 1.24s
+</codeblock>
+
+<p> The <codeph>NDV()</codeph> function stands for <q>number of distinct
+          values</q>, which for performance reasons is an estimate when there
+        are lots of different values in the column, but is precise when the
+        cardinality is less than 16 K. Use <codeph>NDV()</codeph> calls for this
+        kind of exploration rather than <codeph>COUNT(DISTINCT
+            <varname>colname</varname>)</codeph>, because Impala can evaluate
+        multiple <codeph>NDV()</codeph> functions in a single query, but only a
+        single instance of <codeph>COUNT DISTINCT</codeph>. Here we see that
+        there are modest numbers of different airlines, flight numbers, and
+        origin and destination airports. Two things jump out from this query:
+        the number of <codeph>tail_num</codeph> values is much smaller than we
+        might have expected, and there are more destination airports than origin
+        airports. Let's dig further. What we find is that most
+          <codeph>tail_num</codeph> values are <codeph>NULL</codeph>. It looks
+        like this was an experimental column that wasn't filled in accurately.
+        We make a mental note that if we use this data as a starting point,
+        we'll ignore this column. We also find that certain airports are
+        represented in the <codeph>ORIGIN</codeph> column but not the
+          <codeph>DEST</codeph> column; now we know that we cannot rely on the
+        assumption that those sets of airport codes are identical. </p>
+
+<note>
+A slight digression for some performance tuning. Notice how the first
+<codeph>SELECT DISTINCT DEST</codeph> query takes almost 40 seconds.
+We expect all queries on such a small data set, less than 2 GB, to
+take a few seconds at most. The reason is because the expression
+<codeph>NOT IN (SELECT origin FROM airlines_external)</codeph>
+produces an intermediate result set of 123 million rows, then
+runs 123 million comparisons on each data node against the tiny set of destination airports.
+The way the <codeph>NOT IN</codeph> operator works internally means that
+this intermediate result set with 123 million rows might be transmitted
+across the network to each data node in the cluster.
+Applying another <codeph>DISTINCT</codeph> inside the <codeph>NOT IN</codeph>
+subquery means that the intermediate result set is only 340 items,
+resulting in much less network traffic and fewer comparison operations.
+The more efficient query with the added <codeph>DISTINCT</codeph> is approximately 7 times as fast.
+</note>
+
+<codeblock>[localhost:21000] > select ndv(carrier), ndv(flight_num), ndv(tail_num),
+                  >   ndv(origin), ndv(dest) from airlines_external;
++--------------+-----------------+---------------+-------------+-----------+
+| ndv(carrier) | ndv(flight_num) | ndv(tail_num) | ndv(origin) | ndv(dest) |
++--------------+-----------------+---------------+-------------+-----------+
+| 29           | 9086            | 3             | 340         | 347       |
++--------------+-----------------+---------------+-------------+-----------+
+[localhost:21000] > select tail_num, count(*) as howmany from airlines_external
+                  >   group by tail_num;
++----------+-----------+
+| tail_num | howmany   |
++----------+-----------+
+| 715      | 1         |
+| 0        | 406405    |
+| 112      | 6562      |
+| NULL     | 123122001 |
++----------+-----------+
+Fetched 1 row(s) in 5.18s
+[localhost:21000] > select distinct dest from airlines_external
+                  >   where dest not in (select origin from airlines_external);
++------+
+| dest |
++------+
+| LBF  |
+| CBM  |
+| RCA  |
+| SKA  |
+| LAR  |
++------+
+Fetched 5 row(s) in 39.64s
+[localhost:21000] > select distinct dest from airlines_external
+                  >   where dest not in (select distinct origin from airlines_external);
++------+
+| dest |
++------+
+| LBF  |
+| RCA  |
+| CBM  |
+| SKA  |
+| LAR  |
++------+
+Fetched 5 row(s) in 5.59s
+[localhost:21000] > select distinct origin from airlines_external
+                  >   where origin not in (select distinct dest from airlines_external);
+Fetched 0 row(s) in 5.37s
+</codeblock>
+
+<p> Next, we try doing a simple calculation, with results broken down by year.
+        This reveals that some years have no data in the
+          <codeph>AIRTIME</codeph> column. That means we might be able to use
+        that column in queries involving certain date ranges, but we cannot
+        count on it to always be reliable. The question of whether a column
+        contains any <codeph>NULL</codeph> values, and if so what is their
+        number, proportion, and distribution, comes up again and again when
+        doing initial exploration of a data set. </p>
+
+<codeblock>[localhost:21000] > select year, sum(airtime) from airlines_external
+                  >   group by year order by year desc;
++------+--------------+
+| year | sum(airtime) |
++------+--------------+
+| 2008 | 713050445    |
+| 2007 | 748015545    |
+| 2006 | 720372850    |
+| 2005 | 708204026    |
+| 2004 | 714276973    |
+| 2003 | 665706940    |
+| 2002 | 549761849    |
+| 2001 | 590867745    |
+| 2000 | 583537683    |
+| 1999 | 561219227    |
+| 1998 | 538050663    |
+| 1997 | 536991229    |
+| 1996 | 519440044    |
+| 1995 | 513364265    |
+| 1994 | NULL         |
+| 1993 | NULL         |
+| 1992 | NULL         |
+| 1991 | NULL         |
+| 1990 | NULL         |
+| 1989 | NULL         |
+| 1988 | NULL         |
+| 1987 | NULL         |
++------+--------------+
+</codeblock>
+
+<p>
+With the notion of <codeph>NULL</codeph> values in mind, let's come back to the <codeph>TAILNUM</codeph>
+column that we discovered had a lot of <codeph>NULL</codeph>s.
+Let's quantify the <codeph>NULL</codeph> and non-<codeph>NULL</codeph> values in that column for better understanding.
+First, we just count the overall number of rows versus the non-<codeph>NULL</codeph> values in that column.
+That initial result gives the appearance of relatively few non-<codeph>NULL</codeph> values, but we can break
+it down more clearly in a single query.
+Once we have the <codeph>COUNT(*)</codeph> and the <codeph>COUNT(<varname>colname</varname>)</codeph> numbers,
+we can encode that initial query in a <codeph>WITH</codeph> clause, then run a followon query that performs
+multiple arithmetic operations on those values.
+Seeing that only one-third of one percent of all rows have non-<codeph>NULL</codeph> values for the
+<codeph>TAILNUM</codeph> column clearly illustrates that that column is not of much use.
+</p>
+
+<codeblock>[localhost:21000] > select count(*) as 'rows', count(tail_num) as 'non-null tail numbers'
+                  >   from airlines_external;
++-----------+-----------------------+
+| rows      | non-null tail numbers |
++-----------+-----------------------+
+| 123534969 | 412968                |
++-----------+-----------------------+
+Fetched 1 row(s) in 1.51s
+[localhost:21000] > with t1 as
+                  >   (select count(*) as 'rows', count(tail_num) as 'nonnull'
+                  >   from airlines_external)
+                  > select `rows`, `nonnull`, `rows` - `nonnull` as 'nulls',
+                  >   (`nonnull` / `rows`) * 100 as 'percentage non-null'
+                  > from t1;
++-----------+---------+-----------+---------------------+
+| rows      | nonnull | nulls     | percentage non-null |
++-----------+---------+-----------+---------------------+
+| 123534969 | 412968  | 123122001 | 0.3342923897119365  |
++-----------+---------+-----------+---------------------+
+</codeblock>
+
+<p>
+By examining other columns using these techniques, we can form a mental picture of the way data is distributed
+throughout the table, and which columns are most significant for query purposes. For this tutorial, we focus mostly on
+the fields likely to hold discrete values, rather than columns such as <codeph>ACTUAL_ELAPSED_TIME</codeph>
+whose names suggest they hold measurements. We would dig deeper into those columns once we had a clear picture
+of which questions were worthwhile to ask, and what kinds of trends we might look for.
+For the final piece of initial exploration, let's look at the <codeph>YEAR</codeph> column.
+A simple <codeph>GROUP BY</codeph> query shows that it has a well-defined range, a manageable number of
+distinct values, and relatively even distribution of rows across the different years.
+</p>
+
+<codeblock>[localhost:21000] > select min(year), max(year), ndv(year) from airlines_external;
++-----------+-----------+-----------+
+| min(year) | max(year) | ndv(year) |
++-----------+-----------+-----------+
+| 1987      | 2008      | 22        |
++-----------+-----------+-----------+
+Fetched 1 row(s) in 2.03s
+[localhost:21000] > select year, count(*) howmany from airlines_external
+                  >   group by year order by year desc;
++------+---------+
+| year | howmany |
++------+---------+
+| 2008 | 7009728 |
+| 2007 | 7453215 |
+| 2006 | 7141922 |
+| 2005 | 7140596 |
+| 2004 | 7129270 |
+| 2003 | 6488540 |
+| 2002 | 5271359 |
+| 2001 | 5967780 |
+| 2000 | 5683047 |
+| 1999 | 5527884 |
+| 1998 | 5384721 |
+| 1997 | 5411843 |
+| 1996 | 5351983 |
+| 1995 | 5327435 |
+| 1994 | 5180048 |
+| 1993 | 5070501 |
+| 1992 | 5092157 |
+| 1991 | 5076925 |
+| 1990 | 5270893 |
+| 1989 | 5041200 |
+| 1988 | 5202096 |
+| 1987 | 1311826 |
++------+---------+
+Fetched 22 row(s) in 2.13s
+</codeblock>
+
+<p>
+We could go quite far with the data in this initial raw format, just as we downloaded it from the web.
+If the data set proved to be useful and worth persisting in Impala for extensive queries,
+we might want to copy it to an internal table, letting Impala manage the data files and perhaps
+reorganizing a little for higher efficiency.
+In this next stage of the tutorial, we copy the original data into a partitioned table, still in Parquet format.
+Partitioning based on the <codeph>YEAR</codeph> column lets us run queries with clauses such as <codeph>WHERE year = 2001</codeph>
+or <codeph>WHERE year BETWEEN 1989 AND 1999</codeph>, which can dramatically cut down on I/O by
+ignoring all the data from years outside the desired range.
+Rather than reading all the data and then deciding which rows are in the matching years, Impala can
+zero in on only the data files from specific <codeph>YEAR</codeph> partitions.
+To do this, Impala physically reorganizes the data files, putting the rows from each year into
+data files in a separate HDFS directory for each <codeph>YEAR</codeph> value.
+Along the way, we'll also get rid of the <codeph>TAIL_NUM</codeph> column that proved to be almost entirely <codeph>NULL</codeph>.
+</p>
+
+<p>
+The first step is to create a new table with a layout very similar to the original <codeph>AIRLINES_EXTERNAL</codeph> table.
+We'll do that by reverse-engineering a <codeph>CREATE TABLE</codeph> statement for the first table,
+then tweaking it slightly to include a <codeph>PARTITION BY</codeph> clause for <codeph>YEAR</codeph>,
+and excluding the <codeph>TAIL_NUM</codeph> column.
+The <codeph>SHOW CREATE TABLE</codeph> statement gives us the starting point.
+</

<TRUNCATED>


[36/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_datetime_functions.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_datetime_functions.xml b/docs/topics/impala_datetime_functions.xml
new file mode 100644
index 0000000..dddfee6
--- /dev/null
+++ b/docs/topics/impala_datetime_functions.xml
@@ -0,0 +1,2482 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="datetime_functions">
+
+  <title>Impala Date and Time Functions</title>
+  <titlealts audience="PDF"><navtitle>Date and Time Functions</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Dates and Times"/>
+      <data name="Category" value="Querying"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      The underlying Impala data type for date and time data is
+      <codeph><xref href="impala_timestamp.xml#timestamp">TIMESTAMP</xref></codeph>, which has both a date and a
+      time portion. Functions that extract a single field, such as <codeph>hour()</codeph> or
+      <codeph>minute()</codeph>, typically return an integer value. Functions that format the date portion, such as
+      <codeph>date_add()</codeph> or <codeph>to_date()</codeph>, typically return a string value.
+    </p>
+
+    <p>
+      You can also adjust a <codeph>TIMESTAMP</codeph> value by adding or subtracting an <codeph>INTERVAL</codeph>
+      expression. See <xref href="impala_timestamp.xml#timestamp"/> for details. <codeph>INTERVAL</codeph>
+      expressions are also allowed as the second argument for the <codeph>date_add()</codeph> and
+      <codeph>date_sub()</codeph> functions, rather than integers.
+    </p>
+
+    <p rev="2.2.0">
+      Some of these functions are affected by the setting of the
+      <codeph>-use_local_tz_for_unix_timestamp_conversions</codeph> startup flag for the
+      <cmdname>impalad</cmdname> daemon. This setting is off by default, meaning that
+      functions such as <codeph>from_unixtime()</codeph> and <codeph>unix_timestamp()</codeph>
+      consider the input values to always represent the UTC time zone.
+      This setting also applies when you <codeph>CAST()</codeph> a <codeph>BIGINT</codeph>
+      value to <codeph>TIMESTAMP</codeph>, or a <codeph>TIMESTAMP</codeph>
+      value to <codeph>BIGINT</codeph>.
+      When this setting is enabled, these functions and operations convert to and from
+      values representing the local time zone.
+      See <xref href="impala_timestamp.xml#timestamp"/> for details about how
+      Impala handles time zone considerations for the <codeph>TIMESTAMP</codeph> data type.
+    </p>
+
+    <p>
+      <b>Function reference:</b>
+    </p>
+
+    <p>
+      Impala supports the following data and time functions:
+    </p>
+
+<!-- New for 2.3:
+int_months_between
+timeofday
+timestamp_cmp
+months_between
+-->
+
+    <dl>
+      <dlentry rev="1.4.0" id="add_months">
+
+        <dt>
+          <codeph>add_months(timestamp date, int months)</codeph>, <codeph>add_months(timestamp date, bigint
+          months)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">add_months() function</indexterm>
+          <b>Purpose:</b> Returns the specified date and time plus some number of months.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            Same as <codeph><xref href="#datetime_functions/months_add">months_add()</xref></codeph>.
+            Available in Impala 1.4 and higher. For
+            compatibility when porting code with vendor extensions.
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples demonstrate adding months to construct the same
+            day of the month in a different month; how if the current day of the month
+            does not exist in the target month, the last day of that month is substituted;
+            and how a negative argument produces a return value from a previous month.
+          </p>
+<codeblock>
+select now(), add_months(now(), 2);
++-------------------------------+-------------------------------+
+| now()                         | add_months(now(), 2)          |
++-------------------------------+-------------------------------+
+| 2016-05-31 10:47:00.429109000 | 2016-07-31 10:47:00.429109000 |
++-------------------------------+-------------------------------+
+
+select now(), add_months(now(), 1);
++-------------------------------+-------------------------------+
+| now()                         | add_months(now(), 1)          |
++-------------------------------+-------------------------------+
+| 2016-05-31 10:47:14.540226000 | 2016-06-30 10:47:14.540226000 |
++-------------------------------+-------------------------------+
+
+select now(), add_months(now(), -1);
++-------------------------------+-------------------------------+
+| now()                         | add_months(now(), -1)         |
++-------------------------------+-------------------------------+
+| 2016-05-31 10:47:31.732298000 | 2016-04-30 10:47:31.732298000 |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="adddate">
+
+        <dt>
+          <codeph>adddate(timestamp startdate, int days)</codeph>, <codeph>adddate(timestamp startdate, bigint
+          days)</codeph>,
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">adddate() function</indexterm>
+          <b>Purpose:</b> Adds a specified number of days to a <codeph>TIMESTAMP</codeph> value. Similar to
+          <codeph>date_add()</codeph>, but starts with an actual <codeph>TIMESTAMP</codeph> value instead of a
+          string that is converted to a <codeph>TIMESTAMP</codeph>.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples show how to add a number of days to a <codeph>TIMESTAMP</codeph>.
+            The number of days can also be negative, which gives the same effect as the <codeph>subdate()</codeph> function.
+          </p>
+<codeblock>
+select now() as right_now, adddate(now(), 30) as now_plus_30;
++-------------------------------+-------------------------------+
+| right_now                     | now_plus_30                   |
++-------------------------------+-------------------------------+
+| 2016-05-20 10:23:08.640111000 | 2016-06-19 10:23:08.640111000 |
++-------------------------------+-------------------------------+
+
+select now() as right_now, adddate(now(), -15) as now_minus_15;
++-------------------------------+-------------------------------+
+| right_now                     | now_minus_15                  |
++-------------------------------+-------------------------------+
+| 2016-05-20 10:23:38.214064000 | 2016-05-05 10:23:38.214064000 |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="current_timestamp">
+
+        <dt>
+          <codeph>current_timestamp()</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">current_timestamp() function</indexterm>
+          <b>Purpose:</b> Alias for the <codeph>now()</codeph> function.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now(), current_timestamp();
++-------------------------------+-------------------------------+
+| now()                         | current_timestamp()           |
++-------------------------------+-------------------------------+
+| 2016-05-19 16:10:14.237849000 | 2016-05-19 16:10:14.237849000 |
++-------------------------------+-------------------------------+
+
+select current_timestamp() as right_now,            
+  current_timestamp() + interval 3 hours as in_three_hours;
++-------------------------------+-------------------------------+
+| right_now                     | in_three_hours                |
++-------------------------------+-------------------------------+
+| 2016-05-19 16:13:20.017117000 | 2016-05-19 19:13:20.017117000 |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="date_add">
+
+        <dt>
+          <codeph>date_add(timestamp startdate, int days)</codeph>, <codeph>date_add(timestamp startdate,
+          <varname>interval_expression</varname>)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">date_add() function</indexterm>
+          <b>Purpose:</b> Adds a specified number of days to a <codeph>TIMESTAMP</codeph> value.
+          <!-- Found this not to be true in latest release. I think the signature changed way back.
+          The first argument
+          can be a string, which is automatically cast to <codeph>TIMESTAMP</codeph> if it uses the recognized
+          format, as described in <xref href="impala_timestamp.xml#timestamp"/>.
+          -->
+          With an <codeph>INTERVAL</codeph>
+          expression as the second argument, you can calculate a delta value using other units such as weeks,
+          years, hours, seconds, and so on; see <xref href="impala_timestamp.xml#timestamp"/> for details.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following example shows the simplest usage, of adding a specified number of days
+            to a <codeph>TIMESTAMP</codeph> value:
+          </p>
+<codeblock>
+select now() as right_now, date_add(now(), 7) as next_week;
++-------------------------------+-------------------------------+
+| right_now                     | next_week                     |
++-------------------------------+-------------------------------+
+| 2016-05-20 11:03:48.687055000 | 2016-05-27 11:03:48.687055000 |
++-------------------------------+-------------------------------+
+</codeblock>
+
+          <p>
+            The following examples show the shorthand notation of an <codeph>INTERVAL</codeph>
+            expression, instead of specifying the precise number of days.
+            The <codeph>INTERVAL</codeph> notation also lets you work with units smaller than
+            a single day.
+          </p>
+<codeblock>
+select now() as right_now, date_add(now(), interval 3 weeks) as in_3_weeks;
++-------------------------------+-------------------------------+
+| right_now                     | in_3_weeks                    |
++-------------------------------+-------------------------------+
+| 2016-05-20 11:05:39.173331000 | 2016-06-10 11:05:39.173331000 |
++-------------------------------+-------------------------------+
+
+select now() as right_now, date_add(now(), interval 6 hours) as in_6_hours;
++-------------------------------+-------------------------------+
+| right_now                     | in_6_hours                    |
++-------------------------------+-------------------------------+
+| 2016-05-20 11:13:51.492536000 | 2016-05-20 17:13:51.492536000 |
++-------------------------------+-------------------------------+
+</codeblock>
+
+          <p>
+            Like all date/time functions that deal with months, <codeph>date_add()</codeph>
+            handles nonexistent dates past the end of a month by setting the date to the
+            last day of the month. The following example shows how the nonexistent date
+            April 31st is normalized to April 30th:
+          </p>
+<codeblock>
+select date_add(cast('2016-01-31' as timestamp), interval 3 months) as 'april_31st';
++---------------------+
+| april_31st          |
++---------------------+
+| 2016-04-30 00:00:00 |
++---------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="2.0.0" id="date_part">
+
+        <dt>
+          <codeph>date_part(string, timestamp)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">date_part() function</indexterm>
+          <b>Purpose:</b> Similar to
+          <xref href="impala_datetime_functions.xml#datetime_functions/extract"><codeph>EXTRACT()</codeph></xref>,
+          with the argument order reversed. Supports the same date and time units as <codeph>EXTRACT()</codeph>.
+          For compatibility with SQL code containing vendor extensions.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select date_part('year',now()) as current_year;
++--------------+
+| current_year |
++--------------+
+| 2016         |
++--------------+
+
+select date_part('hour',now()) as hour_of_day;
++-------------+
+| hour_of_day |
++-------------+
+| 11          |
++-------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="date_sub">
+
+        <dt>
+          <codeph>date_sub(timestamp startdate, int days)</codeph>, <codeph>date_sub(timestamp startdate,
+          <varname>interval_expression</varname>)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">date_sub() function</indexterm>
+          <b>Purpose:</b> Subtracts a specified number of days from a <codeph>TIMESTAMP</codeph> value.
+          <!-- Found this not to be true in latest release. I think the signature changed way back.
+          The first argument can be a string, which is automatically cast to <codeph>TIMESTAMP</codeph> if it uses the
+          recognized format, as described in <xref href="impala_timestamp.xml#timestamp"/>.
+          -->
+          With an
+          <codeph>INTERVAL</codeph> expression as the second argument, you can calculate a delta value using other
+          units such as weeks, years, hours, seconds, and so on; see <xref href="impala_timestamp.xml#timestamp"/>
+          for details.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following example shows the simplest usage, of subtracting a specified number of days
+            from a <codeph>TIMESTAMP</codeph> value:
+          </p>
+<codeblock>
+select now() as right_now, date_sub(now(), 7) as last_week;
++-------------------------------+-------------------------------+
+| right_now                     | last_week                     |
++-------------------------------+-------------------------------+
+| 2016-05-20 11:21:30.491011000 | 2016-05-13 11:21:30.491011000 |
++-------------------------------+-------------------------------+
+</codeblock>
+          <p>
+            The following examples show the shorthand notation of an <codeph>INTERVAL</codeph>
+            expression, instead of specifying the precise number of days.
+            The <codeph>INTERVAL</codeph> notation also lets you work with units smaller than
+            a single day.
+          </p>
+<codeblock>
+select now() as right_now, date_sub(now(), interval 3 weeks) as 3_weeks_ago;
++-------------------------------+-------------------------------+
+| right_now                     | 3_weeks_ago                   |
++-------------------------------+-------------------------------+
+| 2016-05-20 11:23:05.176953000 | 2016-04-29 11:23:05.176953000 |
++-------------------------------+-------------------------------+
+
+select now() as right_now, date_sub(now(), interval 6 hours) as 6_hours_ago;
++-------------------------------+-------------------------------+
+| right_now                     | 6_hours_ago                   |
++-------------------------------+-------------------------------+
+| 2016-05-20 11:23:35.439631000 | 2016-05-20 05:23:35.439631000 |
++-------------------------------+-------------------------------+
+</codeblock>
+
+          <p>
+            Like all date/time functions that deal with months, <codeph>date_add()</codeph>
+            handles nonexistent dates past the end of a month by setting the date to the
+            last day of the month. The following example shows how the nonexistent date
+            April 31st is normalized to April 30th:
+          </p>
+<codeblock>
+select date_sub(cast('2016-05-31' as timestamp), interval 1 months) as 'april_31st';
++---------------------+
+| april_31st          |
++---------------------+
+| 2016-04-30 00:00:00 |
++---------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="datediff">
+
+        <dt>
+          <codeph>datediff(timestamp enddate, timestamp startdate)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">datediff() function</indexterm>
+          <b>Purpose:</b> Returns the number of days between two <codeph>TIMESTAMP</codeph> values.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            If the first argument represents a later date than the second argument,
+            the return value is positive. If both arguments represent the same date,
+            the return value is zero. The time portions of the <codeph>TIMESTAMP</codeph>
+            values are irrelevant. For example, 11:59 PM on one day and 12:01 on the next
+            day represent a <codeph>datediff()</codeph> of -1 because the date/time values
+            represent different days, even though the <codeph>TIMESTAMP</codeph> values differ by only 2 minutes.
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following example shows how comparing a <q>late</q> value with 
+            an <q>earlier</q> value produces a positive number. In this case,
+            the result is (365 * 5) + 1, because one of the intervening years is
+            a leap year.
+          </p>
+<codeblock>
+select now() as right_now, datediff(now() + interval 5 years, now()) as in_5_years;
++-------------------------------+------------+
+| right_now                     | in_5_years |
++-------------------------------+------------+
+| 2016-05-20 13:43:55.873826000 | 1826       |
++-------------------------------+------------+
+</codeblock>
+          <p>
+            The following examples show how the return value represent the number of days
+            between the associated dates, regardless of the time portion of each <codeph>TIMESTAMP</codeph>.
+            For example, different times on the same day produce a <codeph>date_diff()</codeph> of 0,
+            regardless of which one is earlier or later. But if the arguments represent different dates,
+            <codeph>date_diff()</codeph> returns a non-zero integer value, regardless of the time portions
+            of the dates.
+          </p>
+<codeblock>
+select now() as right_now, datediff(now(), now() + interval 4 hours) as in_4_hours;
++-------------------------------+------------+
+| right_now                     | in_4_hours |
++-------------------------------+------------+
+| 2016-05-20 13:42:05.302747000 | 0          |
++-------------------------------+------------+
+
+select now() as right_now, datediff(now(), now() - interval 4 hours) as 4_hours_ago;
++-------------------------------+-------------+
+| right_now                     | 4_hours_ago |
++-------------------------------+-------------+
+| 2016-05-20 13:42:21.134958000 | 0           |
++-------------------------------+-------------+
+
+select now() as right_now, datediff(now(), now() + interval 12 hours) as in_12_hours;
++-------------------------------+-------------+
+| right_now                     | in_12_hours |
++-------------------------------+-------------+
+| 2016-05-20 13:42:44.765873000 | -1          |
++-------------------------------+-------------+
+
+select now() as right_now, datediff(now(), now() - interval 18 hours) as 18_hours_ago;
++-------------------------------+--------------+
+| right_now                     | 18_hours_ago |
++-------------------------------+--------------+
+| 2016-05-20 13:54:38.829827000 | 1            |
++-------------------------------+--------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="day">
+
+        <dt>
+          <!-- <codeph>day(string date), <ph id="dayofmonth">dayofmonth(string date)</ph></codeph> -->
+          <codeph>day(timestamp date), <ph id="dayofmonth">dayofmonth(timestamp date)</ph></codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">day() function</indexterm>
+          <b>Purpose:</b> Returns the day field from the date portion of a <codeph>TIMESTAMP</codeph>.
+          The value represents the day of the month, therefore is in the range 1-31, or less for
+          months without 31 days.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples show how the day value corresponds to the day
+            of the month, resetting back to 1 at the start of each month.
+          </p>
+<codeblock>
+select now(), day(now());
++-------------------------------+------------+
+| now()                         | day(now()) |
++-------------------------------+------------+
+| 2016-05-20 15:01:51.042185000 | 20         |
++-------------------------------+------------+
+
+select now() + interval 11 days, day(now() + interval 11 days);
++-------------------------------+-------------------------------+
+| now() + interval 11 days      | day(now() + interval 11 days) |
++-------------------------------+-------------------------------+
+| 2016-05-31 15:05:56.843139000 | 31                            |
++-------------------------------+-------------------------------+
+
+select now() + interval 12 days, day(now() + interval 12 days);
++-------------------------------+-------------------------------+
+| now() + interval 12 days      | day(now() + interval 12 days) |
++-------------------------------+-------------------------------+
+| 2016-06-01 15:06:05.074236000 | 1                             |
++-------------------------------+-------------------------------+
+</codeblock>
+          <p>
+            The following examples show how the day value is <codeph>NULL</codeph>
+            for nonexistent dates or misformatted date strings.
+          </p>
+<codeblock>
+-- 2016 is a leap year, so it has a Feb. 29.
+select day('2016-02-29');
++-------------------+
+| day('2016-02-29') |
++-------------------+
+| 29                |
++-------------------+
+
+-- 2015 is not a leap year, so Feb. 29 is nonexistent.
+select day('2015-02-29');
++-------------------+
+| day('2015-02-29') |
++-------------------+
+| NULL              |
++-------------------+
+
+-- A string that does not match the expected YYYY-MM-DD format
+-- produces an invalid TIMESTAMP, causing day() to return NULL.
+select day('2016-02-028');
++--------------------+
+| day('2016-02-028') |
++--------------------+
+| NULL               |
++--------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.2" id="dayname">
+
+        <dt>
+          <codeph>dayname(timestamp date)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">dayname() function</indexterm>
+          <b>Purpose:</b> Returns the day field from a <codeph>TIMESTAMP</codeph> value, converted to the string
+          corresponding to that day name. The range of return values is <codeph>'Sunday'</codeph> to
+          <codeph>'Saturday'</codeph>. Used in report-generating queries, as an alternative to calling
+          <codeph>dayofweek()</codeph> and turning that numeric return value into a string using a
+          <codeph>CASE</codeph> expression.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples show the day name associated with
+            <codeph>TIMESTAMP</codeph> values representing different days.
+          </p>
+<codeblock>
+select now() as right_now,
+  dayofweek(now()) as todays_day_of_week,
+  dayname(now()) as todays_day_name;
++-------------------------------+--------------------+-----------------+
+| right_now                     | todays_day_of_week | todays_day_name |
++-------------------------------+--------------------+-----------------+
+| 2016-05-31 10:57:03.953670000 | 3                  | Tuesday         |
++-------------------------------+--------------------+-----------------+
+
+select now() + interval 1 day as tomorrow,
+  dayname(now() + interval 1 day) as tomorrows_day_name;
++-------------------------------+--------------------+
+| tomorrow                      | tomorrows_day_name |
++-------------------------------+--------------------+
+| 2016-06-01 10:58:53.945761000 | Wednesday          |
++-------------------------------+--------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.1" id="dayofweek">
+
+        <dt>
+          <!-- <codeph>dayofweek(string date)</codeph> -->
+          <codeph>dayofweek(timestamp date)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">dayofweek() function</indexterm>
+          <b>Purpose:</b> Returns the day field from the date portion of a <codeph>TIMESTAMP</codeph>, corresponding to the day of
+          the week. The range of return values is 1 (Sunday) to 7 (Saturday).
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now,
+  dayofweek(now()) as todays_day_of_week,
+  dayname(now()) as todays_day_name;
++-------------------------------+--------------------+-----------------+
+| right_now                     | todays_day_of_week | todays_day_name |
++-------------------------------+--------------------+-----------------+
+| 2016-05-31 10:57:03.953670000 | 3                  | Tuesday         |
++-------------------------------+--------------------+-----------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="dayofyear">
+
+        <dt>
+          <codeph>dayofyear(timestamp date)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">dayofyear() function</indexterm>
+          <b>Purpose:</b> Returns the day field from a <codeph>TIMESTAMP</codeph> value, corresponding to the day
+          of the year. The range of return values is 1 (January 1) to 366 (December 31 of a leap year).
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples show return values from the
+            <codeph>dayofyear()</codeph> function. The same date
+            in different years returns a different day number
+            for all dates after February 28,
+            because 2016 is a leap year while 2015 is not a leap year.
+          </p>
+<codeblock>
+select now() as right_now,
+  dayofyear(now()) as today_day_of_year;
++-------------------------------+-------------------+
+| right_now                     | today_day_of_year |
++-------------------------------+-------------------+
+| 2016-05-31 11:05:48.314932000 | 152               |
++-------------------------------+-------------------+
+
+select now() - interval 1 year as last_year,
+  dayofyear(now() - interval 1 year) as year_ago_day_of_year;
++-------------------------------+----------------------+
+| last_year                     | year_ago_day_of_year |
++-------------------------------+----------------------+
+| 2015-05-31 11:07:03.733689000 | 151                  |
++-------------------------------+----------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="days_add">
+
+        <dt>
+          <codeph>days_add(timestamp startdate, int days)</codeph>, <codeph>days_add(timestamp startdate, bigint
+          days)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">days_add() function</indexterm>
+          <b>Purpose:</b> Adds a specified number of days to a <codeph>TIMESTAMP</codeph> value. Similar to
+          <codeph>date_add()</codeph>, but starts with an actual <codeph>TIMESTAMP</codeph> value instead of a
+          string that is converted to a <codeph>TIMESTAMP</codeph>.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now, days_add(now(), 31) as 31_days_later;
++-------------------------------+-------------------------------+
+| right_now                     | 31_days_later                 |
++-------------------------------+-------------------------------+
+| 2016-05-31 11:12:32.216764000 | 2016-07-01 11:12:32.216764000 |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="days_sub">
+
+        <dt>
+          <codeph>days_sub(timestamp startdate, int days)</codeph>, <codeph>days_sub(timestamp startdate, bigint
+          days)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">days_sub() function</indexterm>
+          <b>Purpose:</b> Subtracts a specified number of days from a <codeph>TIMESTAMP</codeph> value. Similar to
+          <codeph>date_sub()</codeph>, but starts with an actual <codeph>TIMESTAMP</codeph> value instead of a
+          string that is converted to a <codeph>TIMESTAMP</codeph>.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now, days_sub(now(), 31) as 31_days_ago;
++-------------------------------+-------------------------------+
+| right_now                     | 31_days_ago                   |
++-------------------------------+-------------------------------+
+| 2016-05-31 11:13:42.163905000 | 2016-04-30 11:13:42.163905000 |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.4.0" id="extract">
+
+        <dt>
+          <codeph>extract(timestamp, string unit)</codeph><codeph rev="2.0.0">extract(unit FROM timestamp)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">extract() function</indexterm>
+          <b>Purpose:</b> Returns one of the numeric date or time fields from a <codeph>TIMESTAMP</codeph> value.
+          <p>
+            <b>Unit argument:</b> The <codeph>unit</codeph> string can be one of <codeph>year</codeph>,
+            <codeph>month</codeph>, <codeph>day</codeph>, <codeph>hour</codeph>, <codeph>minute</codeph>,
+            <codeph>second</codeph>, or <codeph>millisecond</codeph>. This argument value is case-insensitive.
+          </p>
+          <p rev="2.0.0">
+            In Impala 2.0 and higher, you can use special syntax rather than a regular function call, for
+            compatibility with code that uses the SQL-99 format with the <codeph>FROM</codeph> keyword. With this
+            style, the unit names are identifiers rather than <codeph>STRING</codeph> literals. For example, the
+            following calls are both equivalent:
+<codeblock>extract(year from now());
+extract(now(), "year");
+</codeblock>
+          </p>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            Typically used in <codeph>GROUP BY</codeph> queries to arrange results by hour,
+            day, month, and so on. You can also use this function in an <codeph>INSERT ... SELECT</codeph> into a
+            partitioned table to split up <codeph>TIMESTAMP</codeph> values into individual parts, if the
+            partitioned table has separate partition key columns representing year, month, day, and so on. If you
+            need to divide by more complex units of time, such as by week or by quarter, use the
+            <codeph>TRUNC()</codeph> function instead.
+          </p>
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now,
+  extract(year from now()) as this_year,
+  extract(month from now()) as this_month;
++-------------------------------+-----------+------------+
+| right_now                     | this_year | this_month |
++-------------------------------+-----------+------------+
+| 2016-05-31 11:18:43.310328000 | 2016      | 5          |
++-------------------------------+-----------+------------+
+
+select now() as right_now,
+  extract(day from now()) as this_day,  
+  extract(hour from now()) as this_hour;  
++-------------------------------+----------+-----------+
+| right_now                     | this_day | this_hour |
++-------------------------------+----------+-----------+
+| 2016-05-31 11:19:24.025303000 | 31       | 11        |
++-------------------------------+----------+-----------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="from_unixtime">
+
+        <dt>
+          <codeph>from_unixtime(bigint unixtime[, string format])</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">from_unixtime() function</indexterm>
+          <b>Purpose:</b> Converts the number of seconds from the Unix epoch to the specified time into a string in
+          the local time zone.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/y2k38"/>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            The format string accepts the variations allowed for the <codeph>TIMESTAMP</codeph>
+            data type: date plus time, date by itself, time by itself, and optional fractional seconds for the
+            time. See <xref href="impala_timestamp.xml#timestamp"/> for details.
+          </p>
+          <p rev="1.3.0">
+            Currently, the format string is case-sensitive, especially to distinguish <codeph>m</codeph> for
+            minutes and <codeph>M</codeph> for months. In Impala 1.3 and later, you can switch the order of
+            elements, use alternative separator characters, and use a different number of placeholders for each
+            unit. Adding more instances of <codeph>y</codeph>, <codeph>d</codeph>, <codeph>H</codeph>, and so on
+            produces output strings zero-padded to the requested number of characters. The exception is
+            <codeph>M</codeph> for months, where <codeph>M</codeph> produces a non-padded value such as
+            <codeph>3</codeph>, <codeph>MM</codeph> produces a zero-padded value such as <codeph>03</codeph>,
+            <codeph>MMM</codeph> produces an abbreviated month name such as <codeph>Mar</codeph>, and sequences of
+            4 or more <codeph>M</codeph> are not allowed. A date string including all fields could be
+            <codeph>"yyyy-MM-dd HH:mm:ss.SSSSSS"</codeph>, <codeph>"dd/MM/yyyy HH:mm:ss.SSSSSS"</codeph>,
+            <codeph>"MMM dd, yyyy HH.mm.ss (SSSSSS)"</codeph> or other combinations of placeholders and separator
+            characters.
+          </p>
+          <p conref="../shared/impala_common.xml#common/timezone_conversion_caveat"/>
+          <note rev="1.3.0">
+            <p rev="1.3.0">
+              The more flexible format strings allowed with the built-in functions do not change the rules about
+              using <codeph>CAST()</codeph> to convert from a string to a <codeph>TIMESTAMP</codeph> value. Strings
+              being converted through <codeph>CAST()</codeph> must still have the elements in the specified order and use the specified delimiter
+              characters, as described in <xref href="impala_timestamp.xml#timestamp"/>.
+            </p>
+          </note>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>select from_unixtime(1392394861,"yyyy-MM-dd HH:mm:ss.SSSS");
++-------------------------------------------------------+
+| from_unixtime(1392394861, 'yyyy-mm-dd hh:mm:ss.ssss') |
++-------------------------------------------------------+
+| 2014-02-14 16:21:01.0000                              |
++-------------------------------------------------------+
+
+select from_unixtime(1392394861,"yyyy-MM-dd");
++-----------------------------------------+
+| from_unixtime(1392394861, 'yyyy-mm-dd') |
++-----------------------------------------+
+| 2014-02-14                              |
++-----------------------------------------+
+
+select from_unixtime(1392394861,"HH:mm:ss.SSSS");
++--------------------------------------------+
+| from_unixtime(1392394861, 'hh:mm:ss.ssss') |
++--------------------------------------------+
+| 16:21:01.0000                              |
++--------------------------------------------+
+
+select from_unixtime(1392394861,"HH:mm:ss");
++---------------------------------------+
+| from_unixtime(1392394861, 'hh:mm:ss') |
++---------------------------------------+
+| 16:21:01                              |
++---------------------------------------+</codeblock>
+          <p conref="../shared/impala_common.xml#common/datetime_function_chaining"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="from_utc_timestamp">
+
+        <dt>
+          <codeph>from_utc_timestamp(timestamp, string timezone)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">from_utc_timestamp() function</indexterm>
+          <b>Purpose:</b> Converts a specified UTC timestamp value into the appropriate value for a specified time
+          zone.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p>
+            <b>Usage notes:</b> Often used to translate UTC time zone data stored in a table back to the local
+            date and time for reporting. The opposite of the <codeph>to_utc_timestamp()</codeph> function.
+          </p>
+          <p conref="../shared/impala_common.xml#common/current_timezone_tip"/>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            See discussion of time zones in <xref href="impala_timestamp.xml#timestamp"/>
+            for information about using this function for conversions between the local time zone and UTC.
+          </p>
+          <p>
+            The following example shows how when <codeph>TIMESTAMP</codeph> values representing the UTC time zone
+            are stored in a table, a query can display the equivalent local date and time for a different time zone.
+          </p>
+<codeblock>
+with t1 as (select cast('2016-06-02 16:25:36.116143000' as timestamp) as utc_datetime)
+  select utc_datetime as 'Date/time in Greenwich UK',
+    from_utc_timestamp(utc_datetime, 'PDT')
+      as 'Equivalent in California USA'
+  from t1;
++-------------------------------+-------------------------------+
+| date/time in greenwich uk     | equivalent in california usa  |
++-------------------------------+-------------------------------+
+| 2016-06-02 16:25:36.116143000 | 2016-06-02 09:25:36.116143000 |
++-------------------------------+-------------------------------+
+</codeblock>
+          <p>
+            The following example shows that for a date and time when daylight savings
+            is in effect (<codeph>PDT</codeph>), the UTC time
+            is 7 hours ahead of the local California time; while when daylight savings
+            is not in effect (<codeph>PST</codeph>), the UTC time is 8 hours ahead of
+            the local California time.
+          </p>
+<codeblock>
+select now() as local_datetime,
+  to_utc_timestamp(now(), 'PDT') as utc_datetime;
++-------------------------------+-------------------------------+
+| local_datetime                | utc_datetime                  |
++-------------------------------+-------------------------------+
+| 2016-05-31 11:50:02.316883000 | 2016-05-31 18:50:02.316883000 |
++-------------------------------+-------------------------------+
+
+select '2016-01-05' as local_datetime,
+  to_utc_timestamp('2016-01-05', 'PST') as utc_datetime;
++----------------+---------------------+
+| local_datetime | utc_datetime        |
++----------------+---------------------+
+| 2016-01-05     | 2016-01-05 08:00:00 |
++----------------+---------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="hour">
+
+        <dt>
+          <codeph>hour(timestamp date)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">hour() function</indexterm>
+          <b>Purpose:</b> Returns the hour field from a <codeph>TIMESTAMP</codeph> field.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now, hour(now()) as current_hour;
++-------------------------------+--------------+
+| right_now                     | current_hour |
++-------------------------------+--------------+
+| 2016-06-01 14:14:12.472846000 | 14           |
++-------------------------------+--------------+
+
+select now() + interval 12 hours as 12_hours_from_now,
+  hour(now() + interval 12 hours) as hour_in_12_hours;
++-------------------------------+-------------------+
+| 12_hours_from_now             | hour_in_12_hours  |
++-------------------------------+-------------------+
+| 2016-06-02 02:15:32.454750000 | 2                 |
++-------------------------------+-------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="hours_add">
+
+        <dt>
+          <codeph>hours_add(timestamp date, int hours)</codeph>, <codeph>hours_add(timestamp date, bigint
+          hours)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">hours_add() function</indexterm>
+          <b>Purpose:</b> Returns the specified date and time plus some number of hours.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now,
+  hours_add(now(), 12) as in_12_hours;
++-------------------------------+-------------------------------+
+| right_now                     | in_12_hours                   |
++-------------------------------+-------------------------------+
+| 2016-06-01 14:19:48.948107000 | 2016-06-02 02:19:48.948107000 |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="hours_sub">
+
+        <dt>
+          <codeph>hours_sub(timestamp date, int hours)</codeph>, <codeph>hours_sub(timestamp date, bigint
+          hours)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">hours_sub() function</indexterm>
+          <b>Purpose:</b> Returns the specified date and time minus some number of hours.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now,
+  hours_sub(now(), 18) as 18_hours_ago;
++-------------------------------+-------------------------------+
+| right_now                     | 18_hours_ago                  |
++-------------------------------+-------------------------------+
+| 2016-06-01 14:23:13.868150000 | 2016-05-31 20:23:13.868150000 |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="2.3.0" id="int_months_between">
+
+        <dt>
+          <codeph>int_months_between(timestamp newer, timestamp older)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">int_months_between() function</indexterm>
+          <b>Purpose:</b> Returns the number of months between the date portions of two <codeph>TIMESTAMP</codeph> values,
+          as an <codeph>INT</codeph> representing only the full months that passed.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            Typically used in business contexts, for example to determine whether
+            a specified number of months have passed or whether some end-of-month deadline was reached.
+          </p>
+          <p>
+            The method of determining the number of elapsed months includes some special handling of
+            months with different numbers of days that creates edge cases for dates between the
+            28th and 31st days of certain months. See <codeph>months_between()</codeph> for details.
+            The <codeph>int_months_between()</codeph> result is essentially the <codeph>floor()</codeph>
+            of the <codeph>months_between()</codeph> result.
+          </p>
+          <p>
+            If either value is <codeph>NULL</codeph>, which could happen for example when converting a
+            nonexistent date string such as <codeph>'2015-02-29'</codeph> to a <codeph>TIMESTAMP</codeph>,
+            the result is also <codeph>NULL</codeph>.
+          </p>
+          <p>
+            If the first argument represents an earlier time than the second argument, the result is negative.
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>/* Less than a full month = 0. */
+select int_months_between('2015-02-28', '2015-01-29');
++------------------------------------------------+
+| int_months_between('2015-02-28', '2015-01-29') |
++------------------------------------------------+
+| 0                                              |
++------------------------------------------------+
+
+/* Last day of month to last day of next month = 1. */
+select int_months_between('2015-02-28', '2015-01-31');
++------------------------------------------------+
+| int_months_between('2015-02-28', '2015-01-31') |
++------------------------------------------------+
+| 1                                              |
++------------------------------------------------+
+
+/* Slightly less than 2 months = 1. */
+select int_months_between('2015-03-28', '2015-01-31');
++------------------------------------------------+
+| int_months_between('2015-03-28', '2015-01-31') |
++------------------------------------------------+
+| 1                                              |
++------------------------------------------------+
+
+/* 2 full months (identical days of the month) = 2. */
+select int_months_between('2015-03-31', '2015-01-31');
++------------------------------------------------+
+| int_months_between('2015-03-31', '2015-01-31') |
++------------------------------------------------+
+| 2                                              |
++------------------------------------------------+
+
+/* Last day of month to last day of month-after-next = 2. */
+select int_months_between('2015-03-31', '2015-01-30');
++------------------------------------------------+
+| int_months_between('2015-03-31', '2015-01-30') |
++------------------------------------------------+
+| 2                                              |
++------------------------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="microseconds_add">
+
+        <dt>
+          <codeph>microseconds_add(timestamp date, int microseconds)</codeph>, <codeph>microseconds_add(timestamp
+          date, bigint microseconds)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">microseconds_add() function</indexterm>
+          <b>Purpose:</b> Returns the specified date and time plus some number of microseconds.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now,
+  microseconds_add(now(), 500000) as half_a_second_from_now;
++-------------------------------+-------------------------------+
+| right_now                     | half_a_second_from_now        |
++-------------------------------+-------------------------------+
+| 2016-06-01 14:25:11.455051000 | 2016-06-01 14:25:11.955051000 |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="microseconds_sub">
+
+        <dt>
+          <codeph>microseconds_sub(timestamp date, int microseconds)</codeph>, <codeph>microseconds_sub(timestamp
+          date, bigint microseconds)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">microseconds_sub() function</indexterm>
+          <b>Purpose:</b> Returns the specified date and time minus some number of microseconds.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now,
+  microseconds_sub(now(), 500000) as half_a_second_ago;
++-------------------------------+-------------------------------+
+| right_now                     | half_a_second_ago             |
++-------------------------------+-------------------------------+
+| 2016-06-01 14:26:16.509990000 | 2016-06-01 14:26:16.009990000 |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="IMPALA-1772 CDH-38381 2.6.0" id="millisecond">
+
+        <dt>
+          <codeph>millisecond(timestamp)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">millisecond() function</indexterm>
+          <b>Purpose:</b> Returns the millisecond portion of a <codeph>TIMESTAMP</codeph> value.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/added_in_250"/>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            The millisecond value is truncated, not rounded, if the <codeph>TIMESTAMP</codeph>
+            value contains more than 3 significant digits to the right of the decimal point.
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+252.4 milliseconds truncated to 252.
+
+select now(), millisecond(now());
++-------------------------------+--------------------+
+| now()                         | millisecond(now()) |
++-------------------------------+--------------------+
+| 2016-03-14 22:30:25.252400000 | 252                |
++-------------------------------+--------------------+
+
+761.767 milliseconds truncated to 761.
+
+select now(), millisecond(now());
++-------------------------------+--------------------+
+| now()                         | millisecond(now()) |
++-------------------------------+--------------------+
+| 2016-03-14 22:30:58.761767000 | 761                |
++-------------------------------+--------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="milliseconds_add">
+
+        <dt>
+          <codeph>milliseconds_add(timestamp date, int milliseconds)</codeph>, <codeph>milliseconds_add(timestamp
+          date, bigint milliseconds)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">milliseconds_add() function</indexterm>
+          <b>Purpose:</b> Returns the specified date and time plus some number of milliseconds.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now,
+  milliseconds_add(now(), 1500) as 1_point_5_seconds_from_now;
++-------------------------------+-------------------------------+
+| right_now                     | 1_point_5_seconds_from_now    |
++-------------------------------+-------------------------------+
+| 2016-06-01 14:30:30.067366000 | 2016-06-01 14:30:31.567366000 |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="milliseconds_sub">
+
+        <dt>
+          <codeph>milliseconds_sub(timestamp date, int milliseconds)</codeph>, <codeph>milliseconds_sub(timestamp
+          date, bigint milliseconds)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">milliseconds_sub() function</indexterm>
+          <b>Purpose:</b> Returns the specified date and time minus some number of milliseconds.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now,
+  milliseconds_sub(now(), 1500) as 1_point_5_seconds_ago;
++-------------------------------+-------------------------------+
+| right_now                     | 1_point_5_seconds_ago         |
++-------------------------------+-------------------------------+
+| 2016-06-01 14:30:53.467140000 | 2016-06-01 14:30:51.967140000 |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="minute">
+
+        <dt>
+          <codeph>minute(timestamp date)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">minute() function</indexterm>
+          <b>Purpose:</b> Returns the minute field from a <codeph>TIMESTAMP</codeph> value.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now, minute(now()) as current_minute;
++-------------------------------+----------------+
+| right_now                     | current_minute |
++-------------------------------+----------------+
+| 2016-06-01 14:34:08.051702000 | 34             |
++-------------------------------+----------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="minutes_add">
+
+        <dt>
+          <codeph>minutes_add(timestamp date, int minutes)</codeph>, <codeph>minutes_add(timestamp date, bigint
+          minutes)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">minutes_add() function</indexterm>
+          <b>Purpose:</b> Returns the specified date and time plus some number of minutes.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now, minutes_add(now(), 90) as 90_minutes_from_now;
++-------------------------------+-------------------------------+
+| right_now                     | 90_minutes_from_now           |
++-------------------------------+-------------------------------+
+| 2016-06-01 14:36:04.887095000 | 2016-06-01 16:06:04.887095000 |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="minutes_sub">
+
+        <dt>
+          <codeph>minutes_sub(timestamp date, int minutes)</codeph>, <codeph>minutes_sub(timestamp date, bigint
+          minutes)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">minutes_sub() function</indexterm>
+          <b>Purpose:</b> Returns the specified date and time minus some number of minutes.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now, minutes_sub(now(), 90) as 90_minutes_ago;
++-------------------------------+-------------------------------+
+| right_now                     | 90_minutes_ago                |
++-------------------------------+-------------------------------+
+| 2016-06-01 14:36:32.643061000 | 2016-06-01 13:06:32.643061000 |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="month">
+
+        <dt>
+          <!-- <codeph>month(string date)</codeph> -->
+          <codeph>month(timestamp date)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">month() function</indexterm>
+          <b>Purpose:</b> Returns the month field, represented as an integer, from the date portion of a <codeph>TIMESTAMP</codeph>.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now, month(now()) as current_month;
++-------------------------------+---------------+
+| right_now                     | current_month |
++-------------------------------+---------------+
+| 2016-06-01 14:43:37.141542000 | 6             |
++-------------------------------+---------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="months_add">
+
+        <dt>
+          <codeph>months_add(timestamp date, int months)</codeph>, <codeph>months_add(timestamp date, bigint
+          months)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">months_add() function</indexterm>
+          <b>Purpose:</b> Returns the specified date and time plus some number of months.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following example shows the effects of adding some number of
+            months to a <codeph>TIMESTAMP</codeph> value, using both the
+            <codeph>months_add()</codeph> function and its <codeph>add_months()</codeph>
+            alias. These examples use <codeph>trunc()</codeph> to strip off the time portion
+            and leave just the date.
+          </p>
+<codeblock>
+with t1 as (select trunc(now(), 'dd') as today)
+  select today, months_add(today,1) as next_month from t1;
++---------------------+---------------------+
+| today               | next_month          |
++---------------------+---------------------+
+| 2016-05-19 00:00:00 | 2016-06-19 00:00:00 |
++---------------------+---------------------+
+
+with t1 as (select trunc(now(), 'dd') as today)
+  select today, add_months(today,1) as next_month from t1;
++---------------------+---------------------+
+| today               | next_month          |
++---------------------+---------------------+
+| 2016-05-19 00:00:00 | 2016-06-19 00:00:00 |
++---------------------+---------------------+
+</codeblock>
+          <p>
+            The following examples show how if <codeph>months_add()</codeph>
+            would return a nonexistent date, due to different months having
+            different numbers of days, the function returns a <codeph>TIMESTAMP</codeph>
+            from the last day of the relevant month. For example, adding one month
+            to January 31 produces a date of February 29th in the year 2016 (a leap year),
+            and February 28th in the year 2015 (a non-leap year).
+          </p>
+<codeblock>
+with t1 as (select cast('2016-01-31' as timestamp) as jan_31)
+  select jan_31, months_add(jan_31,1) as feb_31 from t1;
++---------------------+---------------------+
+| jan_31              | feb_31              |
++---------------------+---------------------+
+| 2016-01-31 00:00:00 | 2016-02-29 00:00:00 |
++---------------------+---------------------+
+
+with t1 as (select cast('2015-01-31' as timestamp) as jan_31)
+  select jan_31, months_add(jan_31,1) as feb_31 from t1;
++---------------------+---------------------+
+| jan_31              | feb_31              |
++---------------------+---------------------+
+| 2015-01-31 00:00:00 | 2015-02-28 00:00:00 |
++---------------------+---------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="2.3.0" id="months_between">
+
+        <dt>
+          <codeph>months_between(timestamp newer, timestamp older)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">months_between() function</indexterm>
+          <b>Purpose:</b> Returns the number of months between the date portions of two <codeph>TIMESTAMP</codeph> values.
+          Can include a fractional part representing extra days in addition to the full months
+          between the dates. The fractional component is computed by dividing the difference in days by 31 (regardless of the month).
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            Typically used in business contexts, for example to determine whether
+            a specified number of months have passed or whether some end-of-month deadline was reached.
+          </p>
+          <p>
+            If the only consideration is the number of full months and any fractional value is
+            not significant, use <codeph>int_months_between()</codeph> instead.
+          </p>
+          <p>
+            The method of determining the number of elapsed months includes some special handling of
+            months with different numbers of days that creates edge cases for dates between the
+            28th and 31st days of certain months.
+          </p>
+          <p>
+            If either value is <codeph>NULL</codeph>, which could happen for example when converting a
+            nonexistent date string such as <codeph>'2015-02-29'</codeph> to a <codeph>TIMESTAMP</codeph>,
+            the result is also <codeph>NULL</codeph>.
+          </p>
+          <p>
+            If the first argument represents an earlier time than the second argument, the result is negative.
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples show how dates that are on the same day of the month
+            are considered to be exactly N months apart, even if the months have different
+            numbers of days.
+          </p>
+<codeblock>select months_between('2015-02-28', '2015-01-28');
++--------------------------------------------+
+| months_between('2015-02-28', '2015-01-28') |
++--------------------------------------------+
+| 1                                          |
++--------------------------------------------+
+
+select months_between(now(), now() + interval 1 month);
++-------------------------------------------------+
+| months_between(now(), now() + interval 1 month) |
++-------------------------------------------------+
+| -1                                              |
++-------------------------------------------------+
+
+select months_between(now() + interval 1 year, now());
++------------------------------------------------+
+| months_between(now() + interval 1 year, now()) |
++------------------------------------------------+
+| 12                                             |
++------------------------------------------------+
+</codeblock>
+          <p>
+            The following examples show how dates that are on the last day of the month
+            are considered to be exactly N months apart, even if the months have different
+            numbers of days. For example, from January 28th to February 28th is exactly one
+            month because the day of the month is identical; January 31st to February 28th
+            is exactly one month because in both cases it is the last day of the month;
+            but January 29th or 30th to February 28th is considered a fractional month.
+          </p>
+<codeblock>select months_between('2015-02-28', '2015-01-31');
++--------------------------------------------+
+| months_between('2015-02-28', '2015-01-31') |
++--------------------------------------------+
+| 1                                          |
++--------------------------------------------+
+
+select months_between('2015-02-28', '2015-01-29');
++--------------------------------------------+
+| months_between('2015-02-28', '2015-01-29') |
++--------------------------------------------+
+| 0.967741935483871                          |
++--------------------------------------------+
+
+select months_between('2015-02-28', '2015-01-30');;
++--------------------------------------------+
+| months_between('2015-02-28', '2015-01-30') |
++--------------------------------------------+
+| 0.935483870967742                          |
++--------------------------------------------+
+</codeblock>
+          <p>
+            The following examples show how dates that are not a precise number
+            of months apart result in a fractional return value.
+          </p>
+<codeblock>select months_between('2015-03-01', '2015-01-28');
++--------------------------------------------+
+| months_between('2015-03-01', '2015-01-28') |
++--------------------------------------------+
+| 1.129032258064516                          |
++--------------------------------------------+
+
+select months_between('2015-03-01', '2015-02-28');
++--------------------------------------------+
+| months_between('2015-03-01', '2015-02-28') |
++--------------------------------------------+
+| 0.1290322580645161                         |
++--------------------------------------------+
+
+select months_between('2015-06-02', '2015-05-29');
++--------------------------------------------+
+| months_between('2015-06-02', '2015-05-29') |
++--------------------------------------------+
+| 0.1290322580645161                         |
++--------------------------------------------+
+
+select months_between('2015-03-01', '2015-01-25');
++--------------------------------------------+
+| months_between('2015-03-01', '2015-01-25') |
++--------------------------------------------+
+| 1.225806451612903                          |
++--------------------------------------------+
+
+select months_between('2015-03-01', '2015-02-25');
++--------------------------------------------+
+| months_between('2015-03-01', '2015-02-25') |
++--------------------------------------------+
+| 0.2258064516129032                         |
++--------------------------------------------+
+
+select months_between('2015-02-28', '2015-02-01');
++--------------------------------------------+
+| months_between('2015-02-28', '2015-02-01') |
++--------------------------------------------+
+| 0.8709677419354839                         |
++--------------------------------------------+
+
+select months_between('2015-03-28', '2015-03-01');
++--------------------------------------------+
+| months_between('2015-03-28', '2015-03-01') |
++--------------------------------------------+
+| 0.8709677419354839                         |
++--------------------------------------------+
+</codeblock>
+          <p>
+            The following examples show how the time portion of the <codeph>TIMESTAMP</codeph>
+            values are irrelevant for calculating the month interval. Even the fractional part
+            of the result only depends on the number of full days between the argument values,
+            regardless of the time portion.
+          </p>
+<codeblock>select months_between('2015-05-28 23:00:00', '2015-04-28 11:45:00');
++--------------------------------------------------------------+
+| months_between('2015-05-28 23:00:00', '2015-04-28 11:45:00') |
++--------------------------------------------------------------+
+| 1                                                            |
++--------------------------------------------------------------+
+
+select months_between('2015-03-28', '2015-03-01');
++--------------------------------------------+
+| months_between('2015-03-28', '2015-03-01') |
++--------------------------------------------+
+| 0.8709677419354839                         |
++--------------------------------------------+
+
+select months_between('2015-03-28 23:00:00', '2015-03-01 11:45:00');
++--------------------------------------------------------------+
+| months_between('2015-03-28 23:00:00', '2015-03-01 11:45:00') |
++--------------------------------------------------------------+
+| 0.8709677419354839                                           |
++--------------------------------------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="months_sub">
+
+        <dt>
+          <codeph>months_sub(timestamp date, int months)</codeph>, <codeph>months_sub(timestamp date, bigint
+          months)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">months_sub() function</indexterm>
+          <b>Purpose:</b> Returns the specified date and time minus some number of months.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+with t1 as (select trunc(now(), 'dd') as today)
+  select today, months_sub(today,1) as last_month from t1;
++---------------------+---------------------+
+| today               | last_month          |
++---------------------+---------------------+
+| 2016-06-01 00:00:00 | 2016-05-01 00:00:00 |
++---------------------+---------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="nanoseconds_add">
+
+        <dt>
+          <codeph>nanoseconds_add(timestamp date, int nanoseconds)</codeph>, <codeph>nanoseconds_add(timestamp
+          date, bigint nanoseconds)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">nanoseconds_add() function</indexterm>
+          <b>Purpose:</b> Returns the specified date and time plus some number of nanoseconds.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now, nanoseconds_add(now(), 1) as 1_nanosecond_later;
++-------------------------------+-------------------------------+
+| right_now                     | 1_nanosecond_later            |
++-------------------------------+-------------------------------+
+| 2016-06-01 15:42:00.361026000 | 2016-06-01 15:42:00.361026001 |
++-------------------------------+-------------------------------+
+
+-- 1 billion nanoseconds = 1 second.
+select now() as right_now, nanoseconds_add(now(), 1e9) as 1_second_later;
++-------------------------------+-------------------------------+
+| right_now                     | 1_second_later                |
++-------------------------------+-------------------------------+
+| 2016-06-01 15:42:52.926706000 | 2016-06-01 15:42:53.926706000 |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="nanoseconds_sub">
+
+        <dt>
+          <codeph>nanoseconds_sub(timestamp date, int nanoseconds)</codeph>, <codeph>nanoseconds_sub(timestamp
+          date, bigint nanoseconds)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">nanoseconds_sub() function</indexterm>
+          <b>Purpose:</b> Returns the specified date and time minus some number of nanoseconds.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+<codeblock>
+select now() as right_now, nanoseconds_sub(now(), 1) as 1_nanosecond_earlier;
++-------------------------------+-------------------------------+
+| right_now                     | 1_nanosecond_earlier          |
++-------------------------------+-------------------------------+
+| 2016-06-01 15:44:14.355837000 | 2016-06-01 15:44:14.355836999 |
++-------------------------------+-------------------------------+
+
+-- 1 billion nanoseconds = 1 second.
+select now() as right_now, nanoseconds_sub(now(), 1e9) as 1_second_earlier;
++-------------------------------+-------------------------------+
+| right_now                     | 1_second_earlier              |
++-------------------------------+-------------------------------+
+| 2016-06-01 15:44:54.474929000 | 2016-06-01 15:44:53.474929000 |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="now">
+
+        <dt>
+          <codeph>now()</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">now() function</indexterm>
+          <b>Purpose:</b> Returns the current date and time (in the local time zone) as a
+          <codeph>TIMESTAMP</codeph> value.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            To find a date/time value in the future or the past relative to the current date
+            and time, add or subtract an <codeph>INTERVAL</codeph> expression to the return value of
+            <codeph>now()</codeph>. See <xref href="impala_timestamp.xml#timestamp"/> for examples.
+          </p>
+          <p>
+            To produce a <codeph>TIMESTAMP</codeph> representing the current date and time that can be
+            shared or stored without interoperability problems due to time zone differences, use the
+            <codeph>to_utc_timestamp()</codeph> function and specify the time zone of the server.
+            When <codeph>TIMESTAMP</codeph> data is stored in UTC form, any application that queries
+            those values can convert them to the appropriate local time zone by calling the inverse
+            function, <codeph>from_utc_timestamp()</codeph>.
+          </p>
+          <p conref="../shared/impala_common.xml#common/current_timezone_tip"/>
+          <p>
+            Any references to the <codeph>now()</codeph> function are evaluated at the start of a query.
+            All calls to <codeph>now()</codeph> within the same query return the same value,
+            and the value does not depend on how long the query takes.
+          </p>
+
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as 'Current time in California USA',
+  to_utc_timestamp(now(), 'PDT') as 'Current time in Greenwich UK';
++--------------------------------+-------------------------------+
+| current time in california usa | current time in greenwich uk  |
++--------------------------------+-------------------------------+
+| 2016-06-01 15:52:08.980072000  | 2016-06-01 22:52:08.980072000 |
++--------------------------------+-------------------------------+
+
+select now() as right_now,
+  now() + interval 1 day as tomorrow,
+  now() + interval 1 week - interval 3 hours as almost_a_week_from_now;
++-------------------------------+-------------------------------+-------------------------------+
+| right_now                     | tomorrow                      | almost_a_week_from_now        |
++-------------------------------+-------------------------------+-------------------------------+
+| 2016-06-01 15:55:39.671690000 | 2016-06-02 15:55:39.671690000 | 2016-06-08 12:55:39.671690000 |
++-------------------------------+-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="second">
+
+        <dt>
+          <codeph>second(timestamp date)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">second() function</indexterm>
+          <b>Purpose:</b> Returns the second field from a <codeph>TIMESTAMP</codeph> value.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now,
+  second(now()) as seconds_in_current_minute;
++-------------------------------+---------------------------+
+| right_now                     | seconds_in_current_minute |
++-------------------------------+---------------------------+
+| 2016-06-01 16:03:57.006603000 | 57                        |
++-------------------------------+---------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="seconds_add">
+
+        <dt>
+          <codeph>seconds_add(timestamp date, int seconds)</codeph>, <codeph>seconds_add(timestamp date, bigint
+          seconds)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">seconds_add() function</indexterm>
+          <b>Purpose:</b> Returns the specified date and time plus some number of seconds.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now,
+  seconds_add(now(), 10) as 10_seconds_from_now;
++-------------------------------+-------------------------------+
+| right_now                     | 10_seconds_from_now           |
++-------------------------------+-------------------------------+
+| 2016-06-01 16:05:21.573935000 | 2016-06-01 16:05:31.573935000 |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="seconds_sub">
+
+        <dt>
+          <codeph>seconds_sub(timestamp date, int seconds)</codeph>, <codeph>seconds_sub(timestamp date, bigint
+          seconds)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">seconds_sub() function</indexterm>
+          <b>Purpose:</b> Returns the specified date and time minus some number of seconds.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now,
+  seconds_sub(now(), 10) as 10_seconds_ago;
++-------------------------------+-------------------------------+
+| right_now                     | 10_seconds_ago                |
++-------------------------------+-------------------------------+
+| 2016-06-01 16:06:03.467931000 | 2016-06-01 16:05:53.467931000 |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="subdate">
+
+        <dt>
+          <codeph>subdate(timestamp startdate, int days)</codeph>, <codeph>subdate(timestamp startdate, bigint
+          days)</codeph>,
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">subdate() function</indexterm>
+          <b>Purpose:</b> Subtracts a specified number of days from a <codeph>TIMESTAMP</codeph> value. Similar to
+          <codeph>date_sub()</codeph>, but starts with an actual <codeph>TIMESTAMP</codeph> value instead of a
+          string that is converted to a <codeph>TIMESTAMP</codeph>.
+          <p>
+            <b>Return type:</b> <codeph>timestamp</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples show how to subtract a number of days from a <codeph>TIMESTAMP</codeph>.
+            The number of days can also be negative, which gives the same effect as the <codeph>adddate()</codeph> function.
+          </p>
+<codeblock>
+select now() as right_now, subdate(now(), 30) as now_minus_30;
++-------------------------------+-------------------------------+
+| right_now                     | now_minus_30                  |
++-------------------------------+-------------------------------+
+| 2016-05-20 11:00:15.084991000 | 2016-04-20 11:00:15.084991000 |
++-------------------------------+-------------------------------+
+
+select now() as right_now, subdate(now(), -15) as now_plus_15;
++-------------------------------+-------------------------------+
+| right_now                     | now_plus_15                   |
++-------------------------------+-------------------------------+
+| 2016-05-20 11:00:44.766091000 | 2016-06-04 11:00:44.766091000 |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="2.3.0" id="timeofday">
+
+        <dt>
+          <codeph>timeofday()</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">timeofday() function</indexterm>
+          <b>Purpose:</b> Returns a string representation of the current date and time, according to the time of the local system,
+          including any time zone designation.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+          <p>
+            <b>Usage notes:</b> The result value represents similar information as the
+            <codeph>now()</codeph> function, only as a <codeph>STRING</codeph> type
+            and with somewhat different formatting. For example, the day of the week
+            and the time zone identifier are included. This function is intended
+            primarily for compatibility with SQL code from other systems that
+            also have a <codeph>timeofday()</codeph> function. Prefer to use
+            <codeph>now()</codeph> if practical for any new Impala code.
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples show the format of the <codeph>timeofday()</codeph>
+            return value, illustrate how that value is represented as a <codeph>STRING</codeph>
+            that you can manipulate with string processing functions, and how the format
+            compares with the return value from the <codeph>now()</codeph> function.
+          </p>
+<codeblock>/* Date and time fields in a STRING return value. */
+select timeofday();
++------------------------------+
+| timeofday()                  |
++------------------------------+
+| Tue Sep 01 15:13:18 2015 PDT |
++------------------------------+
+
+/* The return value can be processed by other string functions. */
+select upper(timeofday());
++------------------------------+
+| upper(timeofday())           |
++------------------------------+
+| TUE SEP 01 15:13:38 2015 PDT |
++------------------------------+
+
+/* The TIMEOFDAY() result is formatted differently than NOW(). NOW() returns a TIMESTAMP. */
+select now(), timeofday();
++-------------------------------+------------------------------+
+| now()                         | timeofday()                  |
++-------------------------------+------------------------------+
+| 2015-09-01 15:15:25.930021000 | Tue Sep 01 15:15:25 2015 PDT |
++-------------------------------+------------------------------+
+
+/* You can strip out the time zone field to use in calls to from_utc_timestamp(). */
+select regexp_replace(timeofday(), '.* ([A-Z]+)$', '\\1') as current_timezone;
++------------------+
+| current_timezone |
++------------------+
+| PDT              |
++------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="2.3.0" id="timestamp_cmp">
+
+        <dt>
+          <codeph>timestamp_cmp(timestamp t1, timestamp t2)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">timestamp_cmp() function</indexterm>
+          <b>Purpose:</b> Tests if one <codeph>TIMESTAMP</codeph> value is
+          newer than, older than, or identical to another <codeph>TIMESTAMP</codeph>
+          <p>
+            <b>Return type:</b> <codeph>int</codeph> (either -1, 0, 1, or <codeph>NULL</codeph>)
+          </p>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            <b>Usage notes:</b> A comparison function for <codeph>TIMESTAMP</codeph>
+            values that only tests whether the date and time increases, decreases,
+            or stays the same. Similar to the <codeph>sign()</codeph> function
+            for numeric values.
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples show all the possible return values for <codeph>timestamp_cmp()</codeph>.
+            If the first argument represents a later point in time than the second argument, the result is 1.
+            The amount of the difference is irrelevant, only the fact that one argument is greater than or less than the other.
+            If the first argument represents an earlier point in time than the second argument, the result is -1.
+            If the first and second arguments represent identical points in time, the result is 0.
+            If either argument is <codeph>NULL</codeph>, the result is <codeph>NULL</codeph>.
+          </p>
+<codeblock>/* First argument 'later' than second argument. */
+
+select timestamp_cmp(now() + interval 70 minutes, now())
+  as now_vs_in_70_minutes;
++----------------------+
+| now_vs_in_70_minutes |
++----------------------+
+| 1                    |
++----------------------+
+
+select timestamp_cmp(now() +
+  interval 3 days +
+  interval 5 hours, now())
+  as now_vs_days_from_now;
++----------------------+
+| now_vs_days_from_now |
++----------------------+
+| 1                    |
++----------------------+
+
+/* First argument 'earlier' than second argument. */
+select timestamp_cmp(now(), now() + interval 2 hours)
+  as now_vs_2_hours_ago;
++--------------------+
+| now_vs_2_hours_ago |
++--------------------+
+| -1                 |
++--------------------+
+
+/* Both arguments represent the same point in time. */
+
+select timestamp_cmp(now(), now())
+  as identical_timestamps;
++----------------------+
+| identical_timestamps |
++----------------------+
+| 0                    |
++----------------------+
+
+select timestamp_cmp
+(
+  now() + interval 1 hour,
+  now() + interval 60 minutes
+) as equivalent_date_times;
++-----------------------+
+| equivalent_date_times |
++-----------------------+
+| 0                     |
++-----------------------+
+
+/* Either argument NULL. */
+
+select timestamp_cmp(now(), null)
+  as now_vs_null;
++-------------+
+| now_vs_null |
++-------------+
+| NULL        |
++-------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="to_date">
+
+        <dt>
+          <codeph>to_date(timestamp)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">to_date() function</indexterm>
+          <b>Purpose:</b> Returns a string representation of the date field from a timestamp value.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+select now() as right_now,
+  concat('The date today is ',to_date(now()),'.') as date_announcement;
++-------------------------------+-------------------------------+
+| right_now                     | date_announcement             |
++-------------------------------+-------------------------------+
+| 2016-06-01 16:30:36.890325000 | The date today is 2016-06-01. |
++-------------------------------+-------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="to_utc_timestamp">
+
+        <dt>
+          <codeph>to_utc_timestamp(timestamp, string timezone)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">to_utc_timestamp() function</indexterm>
+          <b>Purpose:</b> Converts a specified timestamp value in a 

<TRUNCATED>


[50/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/impala.ditamap
----------------------------------------------------------------------
diff --git a/docs/impala.ditamap b/docs/impala.ditamap
new file mode 100644
index 0000000..2ddd310
--- /dev/null
+++ b/docs/impala.ditamap
@@ -0,0 +1,1172 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE map PUBLIC "-//OASIS//DTD DITA Map//EN" "map.dtd">
+<map id="impala">
+  <title>Apache Impala (incubating) Guide</title>
+  <topicmeta>
+    <prodinfo audience="PDF" conref="shared/ImpalaVariables.xml#impala_vars/prodinfo_for_html">
+      <prodname/>
+      <vrmlist>
+        <vrm version="version_dlq_gry_sm"/>
+      </vrmlist>
+    </prodinfo>
+    <prodinfo audience="HTML" conref="shared/ImpalaVariables.xml#impala_vars/prodinfo_for_pdf">
+      <prodname/>
+      <vrmlist>
+        <vrm version="version_dlq_gry_sm"/>
+      </vrmlist>
+    </prodinfo>
+  </topicmeta>
+
+<!-- Definitions for substitution variables, particularly for upstream/downstream differences in terminology and versioning. -->
+  <keydef keys="hadoop_distro"><topicmeta><keywords><keyword>CDH</keyword></keywords></topicmeta></keydef>
+  <keydef keys="support_org"><topicmeta><keywords><keyword>Cloudera Support</keyword></keywords></topicmeta></keydef>
+
+<!-- URLs used in external links. Many, perhaps most, to be turned into keydefs to genericize and optionally re-point. -->
+<!--
+http://aws.amazon.com/cli/
+http://aws.amazon.com/s3/
+http://community.cloudera.com/t5/Interactive-Short-cycle-SQL/bd-p/Impala
+http://community.cloudera.com/t5/Release-Announcements/bd-p/RelAnnounce
+http://gethue.com/
+http://github.com/Cloudera/Impala/blob/cdh5-trunk/be/src/testutil/test-udas.cc
+http://github.sf.cloudera.com/CDH/Impala/blob/master/testdata/workloads/functional-query/queries/QueryTest/load-hive-udfs.test
+http://go.cloudera.com/odbc-driver-hive-impala.html
+http://goog-perftools.sourceforge.net/doc/tcmalloc.html
+http://google-glog.googlecode.com/svn/trunk/doc/glog.html
+http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html#Configuration
+http://hadoop.apache.org/docs/r0.19.0/distcp.html
+http://haproxy.1wt.eu/
+http://hbase.apache.org/book/ch08s04.html
+http://jdbc.postgresql.org/download.html
+http://radar.oreilly.com/2013/10/cloudera-impala-bringing-the-sql-and-hadoop-worlds-together.html
+http://s3tools.org/s3cmd
+http://sentry.incubator.apache.org/
+http://shop.oreilly.com/product/0636920033936.do
+http://stat-computing.org/dataexpo/2009/
+http://stat-computing.org/dataexpo/2009/the-data.html
+http://web.mit.edu/Kerberos/krb5-1.8/
+http://www.akadia.com/services/ssh_test_certificate.html
+http://www.askapache.com/online-tools/htpasswd-generator/
+http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH-Version-and-Packaging-Information/CDH-Version-and-Packaging-Information.html
+http://www.cloudera.com/content/cloudera-content/cloudera-docs/Connectors/PDF/Cloudera-ODBC-Driver-for-Impala-Install-Guide.pdf
+http://www.cloudera.com/content/cloudera/en/documentation/impala/latest.html
+http://www.cloudera.com/content/cloudera/en/downloads/connectors/impala/jdbc/impala-jdbc-v2-5-5.html
+http://www.cloudera.com/content/cloudera/en/products-and-services/cdh/impala.html
+http://www.cloudera.com/content/cloudera/en/resources/library/recordedwebinar/impala-real-time-queries-in-hadoop-webinar-slides.html
+http://www.cloudera.com/content/dam/cloudera/Resources/PDF/whitepaper/AWS_Reference_Architecture_Whitepaper.pdf
+http://www.cloudera.com/content/support/en/documentation.html
+http://www.cloudera.com/content/support/en/documentation/cloudera-connectors-documentation/connector-documentation-latest.html
+http://www.cloudera.com/content/support/en/downloads/download-components/download-products.html?productID=3mDesWddUS
+http://www.cloudera.com/content/support/en/downloads/quickstart_vms.html
+http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cdh_admin_performance.html
+http://www.cloudera.com/documentation/other/reference-architecture/PDF/Impala-HA-with-F5-BIG-IP.pdf
+http://www.iodbc.org/dataspace/iodbc/wiki/iODBC/Downloads
+http://www.lzop.org/
+http://www.mysql.com/products/connector/
+http://www.tpc.org/tpcds/
+https://archive.cloudera.com/cdh/3/hadoop/hdfs_permissions_guide.html
+https://archive.cloudera.com/cdh4/cdh/4/hadoop/hadoop-project-dist/hadoop-common/FileSystemShell.html
+https://archive.cloudera.com/cdh4/cdh/4/hadoop/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html
+https://archive.cloudera.com/cdh4/cdh/4/hbase/
+https://archive.cloudera.com/cdh5/
+https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-site/FairScheduler.html#Configuration
+https://archive.cloudera.com/cdh5/cdh/5/hbase/
+https://archive.cloudera.com/gplextras/debian/squeeze/amd64/gplextras/cloudera.list
+https://archive.cloudera.com/gplextras/redhat/5/x86_64/gplextras/
+https://archive.cloudera.com/gplextras/redhat/5/x86_64/gplextras/cloudera-gplextras4.repo
+https://archive.cloudera.com/gplextras/redhat/6/x86_64/gplextras/cloudera-gplextras4.repo
+https://archive.cloudera.com/gplextras/sles/11/x86_64/gplextras/cloudera-gplextras4.repo
+https://archive.cloudera.com/gplextras/ubuntu/lucid/amd64/gplextras/cloudera.list
+https://archive.cloudera.com/gplextras/ubuntu/precise/amd64/gplextras/cloudera.list
+https://archive.cloudera.com/impala/
+https://archive.cloudera.com/impala/redhat/6/x86_64/impala/cloudera-impala.repo
+https://chromium.googlesource.com/breakpad/breakpad/
+https://code.google.com/p/re2/
+https://community.emc.com/docs/DOC-39522
+https://console.aws.amazon.com/s3/home
+https://downloads.cloudera.com/impala-jdbc/impala-jdbc-0.5-2.zip
+https://github.com/cloudera/Impala/blob/cdh5-trunk/tests/query_test/test_udfs.py
+https://github.com/cloudera/Impala/blob/master/be/src/exprs/timezone_db.cc
+https://github.com/cloudera/impala-tpcds-kit
+https://github.com/cloudera/impala-udf-samples
+https://github.com/cloudera/impala-udf-samples/blob/master/uda-sample.cc
+https://github.com/cloudera/impala-udf-samples/blob/master/udf-sample.h
+https://github.com/cloudera/impala/tree/master/be/src/udf_samples
+https://github.com/onefoursix/Cloudera-Impala-JDBC-Example
+https://groups.google.com/a/cloudera.org/forum/?fromgroups#!forum/impala-user
+https://issues.apache.org/jira/browse/HIVE-3596
+https://issues.apache.org/jira/browse/HIVE-656
+https://issues.apache.org/jira/browse/HIVE-8648
+https://issues.apache.org/jira/browse/HIVE-9866
+https://issues.cloudera.org/
+https://issues.cloudera.org/browse/
+https://issues.cloudera.org/browse/CDH-20513
+https://issues.cloudera.org/browse/CDH-41605
+https://issues.cloudera.org/browse/IMPALA
+https://issues.cloudera.org/issues/?jql=%20type%20%3D%20bug%20and%20project%20%3D%20IMPALA%20AND%20resolution%20%3D%20fixed%20AND%20affectedVersion%20!%3D%20%22Impala%202.5.0%22%20AND%20fixVersion%20%3D%20%22Impala%202.5.0%22%20and%20not%20labels%20%3D%20broken-build%20order%20by%20priority%20desc
+https://issues.cloudera.org/issues/?jql=%20type%20%3D%20bug%20and%20project%20%3D%20IMPALA%20AND%20resolution%20%3D%20fixed%20AND%20affectedVersion%20!%3D%20%22Impala%202.6.0%22%20AND%20fixVersion%20%3D%20%22Impala%202.6.0%22%20and%20not%20labels%20%3D%20broken-build%20order%20by%20priority%20desc
+https://issues.cloudera.org/issues/?jql=%20type%20%3D%20bug%20and%20project%20%3D%20IMPALA%20AND%20resolution%20%3D%20fixed%20AND%20affectedVersion%20!%3D%20%22Impala%202.7.0%22%20AND%20fixVersion%20%3D%20%22Impala%202.7.0%22%20and%20not%20labels%20%3D%20broken-build%20order%20by%20priority%20desc
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+1.0.1%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+1.1%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+1.1.1%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+1.2%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+1.2.1%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+1.2.2%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+1.2.3%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+1.2.4%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+1.3%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+1.3.1%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+1.4%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+1.4.1%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+1.4.2%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+2.0%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+2.0.1%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+2.0.2%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+2.0.3%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+2.0.4%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+2.0.5%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+2.1%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+2.1.1%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+2.1.2%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project%3Dimpala+and+fixVersion%3D%22Impala+2.2%22+and+resolution%3D%22Fixed%22
+https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&amp;jqlQuery=project+%3D+impala+AND+resolution+%3D+Fixed+AND+fixVersion+%3D+%22Impala+1.0%22+ORDER+BY+key+ASC%2C+assignee+ASC%2C+priority+DESC
+-->
+
+<!-- Links to Wikipedia pages for background on industry terminology. -->
+  <keydef href="http://en.wikipedia.org/wiki/.htpasswd" scope="external" format="html" keys="wiki_htpasswd"/>
+  <keydef href="http://en.wikipedia.org/wiki/Coordinated_Universal_Time" scope="external" format="html" keys="wiki_utc"/>
+  <keydef href="http://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function" scope="external" format="html" keys="wiki_fnv"/>
+  <keydef href="http://en.wikipedia.org/wiki/Memoization" scope="external" format="html" keys="wiki_memoization"/>
+  <keydef href="http://en.wikipedia.org/wiki/Standard_deviation" scope="external" format="html" keys="wiki_standard_deviation"/>
+  <keydef href="http://en.wikipedia.org/wiki/Transport_Layer_Security" scope="external" format="html" keys="wiki_tls"/>
+  <keydef href="http://en.wikipedia.org/wiki/Variance" scope="external" format="html" keys="wiki_variance"/>
+  <keydef href="https://en.wikipedia.org/wiki/Circular_shift" scope="external" format="html" keys="wiki_circular_shift"/>
+  <keydef href="https://en.wikipedia.org/wiki/Double-precision_floating-point_format" scope="external" format="html" keys="wiki_double_precision"/>
+  <keydef href="https://en.wikipedia.org/wiki/E_(mathematical_constant" scope="external" format="html" keys="wiki_e"/>
+  <keydef href="https://en.wikipedia.org/wiki/Factorial" scope="external" format="html" keys="wiki_factorial"/>
+  <keydef href="https://en.wikipedia.org/wiki/Hamming_weight" scope="external" format="html" keys="wiki_hamming_weight"/>
+  <keydef href="https://en.wikipedia.org/wiki/Hash_join" scope="external" format="html" keys="wiki_hash_join"/>
+  <keydef href="https://en.wikipedia.org/wiki/Logical_shift" scope="external" format="html" keys="wiki_logical_shift"/>
+  <keydef href="https://en.wikipedia.org/wiki/Natural_logarithm" scope="external" format="html" keys="wiki_natural_logarithm"/>
+  <keydef href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format" scope="external" format="html" keys="wiki_single_precision"/>
+  <keydef href="https://en.wikipedia.org/wiki/Two%27s_complement" scope="external" format="html" keys="wiki_twos_complement"/>
+  <keydef href="https://en.wikipedia.org/wiki/Universally_unique_identifier" scope="external" format="html" keys="wiki_uuid"/>
+
+<!-- Links to Cloudera doc pages. -->
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/admin_llama.html" scope="external" format="html" keys="admin_llama"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_hag_cdh_other_ha.html" scope="external" format="html" keys="cdh_hag_cdh_other_ha"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_hbase_installation.html" scope="external" format="html" keys="cdh_ig_hbase_installation"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_hdfs_caching.html" scope="external" format="html" keys="cdh_ig_hdfs_caching"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_hive_installation.html" scope="external" format="html" keys="cdh_ig_hive_installation"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_hive_jdbc_install.html" scope="external" format="html" keys="cdh_ig_hive_jdbc_install"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_llama_installation.html" scope="external" format="html" keys="cdh_ig_llama_installation"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_parquet.html" scope="external" format="html" keys="cdh_ig_parquet"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_yarn_cluster_deploy.html" scope="external" format="html" keys="cdh_ig_yarn_cluster_deploy"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_55.html#fixed_issues_554" scope="external" format="html" keys="cdh_rn_fixed_in_55"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_552.html" scope="external" format="html" keys="cdh_rn_fixed_in_552"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_560.html" scope="external" format="html" keys="cdh_rn_fixed_in_560"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_cdh5_hadoop_security.html" scope="external" format="html" keys="cdh_sg_cdh5_hadoop_security"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_hive_security.html" scope="external" format="html" keys="cdh_sg_hive_security"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_kerberos_prin_keytab_deploy.html" scope="external" format="html" keys="cdh_sg_kerberos_prin_keytab_deploy"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_dg_logs.html" scope="external" format="html" keys="cm_dg_logs"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_dg_service_monitoring.html" scope="external" format="html" keys="cm_dg_service_monitoring"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_gpl_extras.html" scope="external" format="html" keys="cm_ig_install_gpl_extras"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_impala.html" scope="external" format="html" keys="cm_ig_install_impala"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_client_config.html" scope="external" format="html" keys="cm_mc_client_config"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_impala_service.html" scope="external" format="html" keys="cm_mc_impala_service"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_isilon_service.html" scope="external" format="html" keys="cm_mc_isilon_service"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_managing_resources.html" scope="external" format="html" keys="cm_mc_managing_resources"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_managing_roles.html" scope="external" format="html" keys="cm_mc_managing_roles"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_resource_pools.html" scope="external" format="html" keys="cm_mc_resource_pools"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_service_pools.html" scope="external" format="html" keys="cm_mc_service_pools"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_start_stop_service.html" scope="external" format="html" keys="cm_mc_start_stop_service"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_upgrading_cdh.html" scope="external" format="html" keys="cm_mc_upgrading_cdh"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_yarn_service.html" scope="external" format="html" keys="cm_mc_yarn_service"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_metrics_impala.html" scope="external" format="html" keys="cm_metrics_impala"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_openssl_jks.html" scope="external" format="html" keys="cm_sg_openssl_jks"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_sentry_service.html" scope="external" format="html" keys="cm_sg_sentry_service"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cn_iu_lineage.html" scope="external" format="html" keys="cn_iu_lineage"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/cn_iu_service_audit.html" scope="external" format="html" keys="cn_iu_service_audit"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/datamgmt_impala_lineage_log.html" scope="external" format="html" keys="datamgmt_impala_lineage_log"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/impala.html" scope="external" format="html" keys="impala"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/installation.html" scope="external" format="html" keys="installation"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/pcm_impala.html" scope="external" format="html" keys="pcm_impala"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/rg_vd.html" scope="external" format="html" keys="rg_vd"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/sg_authentication.html" scope="external" format="html" keys="sg_authentication"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/latest/topics/sg_redaction.html" scope="external" format="html" keys="sg_redaction"/>
+  <keydef href="http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_fixed_in_51.html" scope="external" format="html" keys="cdh_rn_fixed_in_51"/>
+  <keydef href="https://www.cloudera.com/documentation/enterprise/latest/topics/sg_hive_sql.html" scope="external" format="html" keys="sg_hive_sql"/>
+  <keydef href="https://www.cloudera.com/documentation/enterprise/latest/topics/cm_metrics_impala.html" scope="external" format="html" keys="cm_metrics_impala"/>
+
+<!-- Links to pages on Hive wiki. -->
+  <keydef href="https://cwiki.apache.org/confluence/display/Hive/AvroSerDe" scope="external" format="html" keys="hive_avroserde"/>
+  <keydef href="https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration" scope="external" format="html" keys="hive_hbaseintegration"/>
+  <keydef href="https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Keywords,Non-reservedKeywordsandReservedKeywords" scope="external" format="html" keys="hive_reserved_words"/>
+  <keydef href="https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LZO" scope="external" format="html" keys="hive_lzo"/>
+  <keydef href="https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF" scope="external" format="html" keys="hive_udf"/>
+  <keydef href="https://cwiki.apache.org/confluence/display/Hive/StatsDev" scope="external" format="html" keys="hive_statsdev"/>
+
+<!-- Pointers to blog posts. -->
+  <keydef href="http://blog.cloudera.com/blog/2012/03/hbase-hadoop-xceivers/" scope="external" format="html" keys="hbase-hadoop-xceivers"/>
+  <keydef href="http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-queries-in-apache-hadoop-for-real" scope="external" format="html" keys="cloudera-impala-real-time-queries-in-apache-hadoop-for-real"/>
+  <keydef href="http://blog.cloudera.com/blog/2013/07/how-hiveserver2-brings-security-and-concurrency-to-apache-hive/" scope="external" format="html" keys="how-hiveserver2-brings-security-and-concurrency-to-apache-hive"/>
+  <keydef href="http://blog.cloudera.com/blog/2013/09/whats-next-for-impala-after-release-1-1/" scope="external" format="html" keys="whats-next-for-impala-after-release-1-1"/>
+  <keydef href="http://blog.cloudera.com/blog/2014/01/impala-performance-dbms-class-speed/" scope="external" format="html" keys="impala-performance-dbms-class-speed"/>
+
+<!-- Pointers to more recent external Impala JIRA URLs. Might need to rewrite all these later for Apache URLs. -->
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1003" scope="external" format="html" keys="IMPALA-1003"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1019" scope="external" format="html" keys="IMPALA-1019"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1024" scope="external" format="html" keys="IMPALA-1024"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1046" scope="external" format="html" keys="IMPALA-1046"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1067" scope="external" format="html" keys="IMPALA-1067"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1079" scope="external" format="html" keys="IMPALA-1079"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1083" scope="external" format="html" keys="IMPALA-1083"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1093" scope="external" format="html" keys="IMPALA-1093"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1101" scope="external" format="html" keys="IMPALA-1101"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1102" scope="external" format="html" keys="IMPALA-1102"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1104" scope="external" format="html" keys="IMPALA-1104"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1105" scope="external" format="html" keys="IMPALA-1105"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1107" scope="external" format="html" keys="IMPALA-1107"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1109" scope="external" format="html" keys="IMPALA-1109"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1118" scope="external" format="html" keys="IMPALA-1118"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1120" scope="external" format="html" keys="IMPALA-1120"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1121" scope="external" format="html" keys="IMPALA-1121"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1123" scope="external" format="html" keys="IMPALA-1123"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1130" scope="external" format="html" keys="IMPALA-1130"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1131" scope="external" format="html" keys="IMPALA-1131"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1133" scope="external" format="html" keys="IMPALA-1133"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1136" scope="external" format="html" keys="IMPALA-1136"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1139" scope="external" format="html" keys="IMPALA-1139"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1142" scope="external" format="html" keys="IMPALA-1142"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1149" scope="external" format="html" keys="IMPALA-1149"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1165" scope="external" format="html" keys="IMPALA-1165"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1168" scope="external" format="html" keys="IMPALA-1168"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1170" scope="external" format="html" keys="IMPALA-1170"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1194" scope="external" format="html" keys="IMPALA-1194"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1236" scope="external" format="html" keys="IMPALA-1236"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1279" scope="external" format="html" keys="IMPALA-1279"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-13" scope="external" format="html" keys="IMPALA-13"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1305" scope="external" format="html" keys="IMPALA-1305"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1334" scope="external" format="html" keys="IMPALA-1334"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1353" scope="external" format="html" keys="IMPALA-1353"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1376" scope="external" format="html" keys="IMPALA-1376"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1381" scope="external" format="html" keys="IMPALA-1381"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-14" scope="external" format="html" keys="IMPALA-14"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1416" scope="external" format="html" keys="IMPALA-1416"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1420" scope="external" format="html" keys="IMPALA-1420"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-144" scope="external" format="html" keys="IMPALA-144"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1453" scope="external" format="html" keys="IMPALA-1453"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1455" scope="external" format="html" keys="IMPALA-1455"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1459" scope="external" format="html" keys="IMPALA-1459"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1471" scope="external" format="html" keys="IMPALA-1471"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1475" scope="external" format="html" keys="IMPALA-1475"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1476" scope="external" format="html" keys="IMPALA-1476"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1477" scope="external" format="html" keys="IMPALA-1477"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1480" scope="external" format="html" keys="IMPALA-1480"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1483" scope="external" format="html" keys="IMPALA-1483"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1493" scope="external" format="html" keys="IMPALA-1493"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-15" scope="external" format="html" keys="IMPALA-15"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1501" scope="external" format="html" keys="IMPALA-1501"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1508" scope="external" format="html" keys="IMPALA-1508"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1519" scope="external" format="html" keys="IMPALA-1519"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1524" scope="external" format="html" keys="IMPALA-1524"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1526" scope="external" format="html" keys="IMPALA-1526"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1528" scope="external" format="html" keys="IMPALA-1528"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1535" scope="external" format="html" keys="IMPALA-1535"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1553" scope="external" format="html" keys="IMPALA-1553"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1556" scope="external" format="html" keys="IMPALA-1556"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1559" scope="external" format="html" keys="IMPALA-1559"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1562" scope="external" format="html" keys="IMPALA-1562"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1568" scope="external" format="html" keys="IMPALA-1568"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1578" scope="external" format="html" keys="IMPALA-1578"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1586" scope="external" format="html" keys="IMPALA-1586"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1588" scope="external" format="html" keys="IMPALA-1588"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1599" scope="external" format="html" keys="IMPALA-1599"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-16" scope="external" format="html" keys="IMPALA-16"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1606" scope="external" format="html" keys="IMPALA-1606"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1607" scope="external" format="html" keys="IMPALA-1607"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1611" scope="external" format="html" keys="IMPALA-1611"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1615" scope="external" format="html" keys="IMPALA-1615"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1619" scope="external" format="html" keys="IMPALA-1619"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1622" scope="external" format="html" keys="IMPALA-1622"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1623" scope="external" format="html" keys="IMPALA-1623"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1646" scope="external" format="html" keys="IMPALA-1646"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1651" scope="external" format="html" keys="IMPALA-1651"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1652" scope="external" format="html" keys="IMPALA-1652"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1654" scope="external" format="html" keys="IMPALA-1654"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1658" scope="external" format="html" keys="IMPALA-1658"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1668" scope="external" format="html" keys="IMPALA-1668"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-167" scope="external" format="html" keys="IMPALA-167"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1674" scope="external" format="html" keys="IMPALA-1674"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1675" scope="external" format="html" keys="IMPALA-1675"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1683" scope="external" format="html" keys="IMPALA-1683"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1705" scope="external" format="html" keys="IMPALA-1705"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1711" scope="external" format="html" keys="IMPALA-1711"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1712" scope="external" format="html" keys="IMPALA-1712"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1730" scope="external" format="html" keys="IMPALA-1730"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1737" scope="external" format="html" keys="IMPALA-1737"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1738" scope="external" format="html" keys="IMPALA-1738"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1740" scope="external" format="html" keys="IMPALA-1740"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1746" scope="external" format="html" keys="IMPALA-1746"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1748" scope="external" format="html" keys="IMPALA-1748"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1755" scope="external" format="html" keys="IMPALA-1755"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1756" scope="external" format="html" keys="IMPALA-1756"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1771" scope="external" format="html" keys="IMPALA-1771"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1772" scope="external" format="html" keys="IMPALA-1772"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1776" scope="external" format="html" keys="IMPALA-1776"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1787" scope="external" format="html" keys="IMPALA-1787"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1792" scope="external" format="html" keys="IMPALA-1792"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1794" scope="external" format="html" keys="IMPALA-1794"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-18" scope="external" format="html" keys="IMPALA-18"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1801" scope="external" format="html" keys="IMPALA-1801"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1805" scope="external" format="html" keys="IMPALA-1805"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1808" scope="external" format="html" keys="IMPALA-1808"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1821" scope="external" format="html" keys="IMPALA-1821"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-183" scope="external" format="html" keys="IMPALA-183"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-185" scope="external" format="html" keys="IMPALA-185"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1854" scope="external" format="html" keys="IMPALA-1854"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1860" scope="external" format="html" keys="IMPALA-1860"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1862" scope="external" format="html" keys="IMPALA-1862"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1865" scope="external" format="html" keys="IMPALA-1865"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1878" scope="external" format="html" keys="IMPALA-1878"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1881" scope="external" format="html" keys="IMPALA-1881"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1888" scope="external" format="html" keys="IMPALA-1888"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1898" scope="external" format="html" keys="IMPALA-1898"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-190" scope="external" format="html" keys="IMPALA-190"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1900" scope="external" format="html" keys="IMPALA-1900"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1917" scope="external" format="html" keys="IMPALA-1917"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1919" scope="external" format="html" keys="IMPALA-1919"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-192" scope="external" format="html" keys="IMPALA-192"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1929" scope="external" format="html" keys="IMPALA-1929"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1934" scope="external" format="html" keys="IMPALA-1934"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1947" scope="external" format="html" keys="IMPALA-1947"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1949" scope="external" format="html" keys="IMPALA-1949"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1952" scope="external" format="html" keys="IMPALA-1952"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1963" scope="external" format="html" keys="IMPALA-1963"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1968" scope="external" format="html" keys="IMPALA-1968"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1983" scope="external" format="html" keys="IMPALA-1983"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-1987" scope="external" format="html" keys="IMPALA-1987"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-20" scope="external" format="html" keys="IMPALA-20"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2005" scope="external" format="html" keys="IMPALA-2005"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2008" scope="external" format="html" keys="IMPALA-2008"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2016" scope="external" format="html" keys="IMPALA-2016"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2018" scope="external" format="html" keys="IMPALA-2018"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-204" scope="external" format="html" keys="IMPALA-204"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2048" scope="external" format="html" keys="IMPALA-2048"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2064" scope="external" format="html" keys="IMPALA-2064"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2065" scope="external" format="html" keys="IMPALA-2065"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2069" scope="external" format="html" keys="IMPALA-2069"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2070" scope="external" format="html" keys="IMPALA-2070"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2086" scope="external" format="html" keys="IMPALA-2086"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2088" scope="external" format="html" keys="IMPALA-2088"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2089" scope="external" format="html" keys="IMPALA-2089"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2090" scope="external" format="html" keys="IMPALA-2090"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2093" scope="external" format="html" keys="IMPALA-2093"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2111" scope="external" format="html" keys="IMPALA-2111"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2113" scope="external" format="html" keys="IMPALA-2113"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2125" scope="external" format="html" keys="IMPALA-2125"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2130" scope="external" format="html" keys="IMPALA-2130"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2133" scope="external" format="html" keys="IMPALA-2133"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2136" scope="external" format="html" keys="IMPALA-2136"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2141" scope="external" format="html" keys="IMPALA-2141"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2144" scope="external" format="html" keys="IMPALA-2144"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2147" scope="external" format="html" keys="IMPALA-2147"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2154" scope="external" format="html" keys="IMPALA-2154"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2161" scope="external" format="html" keys="IMPALA-2161"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2165" scope="external" format="html" keys="IMPALA-2165"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2168" scope="external" format="html" keys="IMPALA-2168"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-217" scope="external" format="html" keys="IMPALA-217"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2178" scope="external" format="html" keys="IMPALA-2178"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2179" scope="external" format="html" keys="IMPALA-2179"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2180" scope="external" format="html" keys="IMPALA-2180"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2184" scope="external" format="html" keys="IMPALA-2184"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2192" scope="external" format="html" keys="IMPALA-2192"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2196" scope="external" format="html" keys="IMPALA-2196"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2199" scope="external" format="html" keys="IMPALA-2199"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2201" scope="external" format="html" keys="IMPALA-2201"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2203" scope="external" format="html" keys="IMPALA-2203"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2204" scope="external" format="html" keys="IMPALA-2204"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2213" scope="external" format="html" keys="IMPALA-2213"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2215" scope="external" format="html" keys="IMPALA-2215"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2216" scope="external" format="html" keys="IMPALA-2216"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2225" scope="external" format="html" keys="IMPALA-2225"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2226" scope="external" format="html" keys="IMPALA-2226"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2233" scope="external" format="html" keys="IMPALA-2233"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2239" scope="external" format="html" keys="IMPALA-2239"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2249" scope="external" format="html" keys="IMPALA-2249"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2251" scope="external" format="html" keys="IMPALA-2251"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2252" scope="external" format="html" keys="IMPALA-2252"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2257" scope="external" format="html" keys="IMPALA-2257"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2264" scope="external" format="html" keys="IMPALA-2264"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2266" scope="external" format="html" keys="IMPALA-2266"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2270" scope="external" format="html" keys="IMPALA-2270"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2273" scope="external" format="html" keys="IMPALA-2273"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2284" scope="external" format="html" keys="IMPALA-2284"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-231" scope="external" format="html" keys="IMPALA-231"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2314" scope="external" format="html" keys="IMPALA-2314"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2319" scope="external" format="html" keys="IMPALA-2319"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2336" scope="external" format="html" keys="IMPALA-2336"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-234" scope="external" format="html" keys="IMPALA-234"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2344" scope="external" format="html" keys="IMPALA-2344"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2348" scope="external" format="html" keys="IMPALA-2348"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2357" scope="external" format="html" keys="IMPALA-2357"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2364" scope="external" format="html" keys="IMPALA-2364"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2365" scope="external" format="html" keys="IMPALA-2365"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2369" scope="external" format="html" keys="IMPALA-2369"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2375" scope="external" format="html" keys="IMPALA-2375"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2378" scope="external" format="html" keys="IMPALA-2378"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-238" scope="external" format="html" keys="IMPALA-238"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2382" scope="external" format="html" keys="IMPALA-2382"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2419" scope="external" format="html" keys="IMPALA-2419"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2422" scope="external" format="html" keys="IMPALA-2422"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-243" scope="external" format="html" keys="IMPALA-243"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2440" scope="external" format="html" keys="IMPALA-2440"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2446" scope="external" format="html" keys="IMPALA-2446"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2473" scope="external" format="html" keys="IMPALA-2473"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2499" scope="external" format="html" keys="IMPALA-2499"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2514" scope="external" format="html" keys="IMPALA-2514"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-252" scope="external" format="html" keys="IMPALA-252"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2533" scope="external" format="html" keys="IMPALA-2533"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2535" scope="external" format="html" keys="IMPALA-2535"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2538" scope="external" format="html" keys="IMPALA-2538"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2552" scope="external" format="html" keys="IMPALA-2552"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2558" scope="external" format="html" keys="IMPALA-2558"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2559" scope="external" format="html" keys="IMPALA-2559"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2560" scope="external" format="html" keys="IMPALA-2560"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-257" scope="external" format="html" keys="IMPALA-257"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2591" scope="external" format="html" keys="IMPALA-2591"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2592" scope="external" format="html" keys="IMPALA-2592"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2598" scope="external" format="html" keys="IMPALA-2598"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2599" scope="external" format="html" keys="IMPALA-2599"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2603" scope="external" format="html" keys="IMPALA-2603"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2612" scope="external" format="html" keys="IMPALA-2612"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2614" scope="external" format="html" keys="IMPALA-2614"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2641" scope="external" format="html" keys="IMPALA-2641"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2643" scope="external" format="html" keys="IMPALA-2643"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2647" scope="external" format="html" keys="IMPALA-2647"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2648" scope="external" format="html" keys="IMPALA-2648"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2649" scope="external" format="html" keys="IMPALA-2649"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-266" scope="external" format="html" keys="IMPALA-266"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2660" scope="external" format="html" keys="IMPALA-2660"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2664" scope="external" format="html" keys="IMPALA-2664"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2665" scope="external" format="html" keys="IMPALA-2665"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-267" scope="external" format="html" keys="IMPALA-267"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2681" scope="external" format="html" keys="IMPALA-2681"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2683" scope="external" format="html" keys="IMPALA-2683"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2686" scope="external" format="html" keys="IMPALA-2686"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2688" scope="external" format="html" keys="IMPALA-2688"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2695" scope="external" format="html" keys="IMPALA-2695"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2696" scope="external" format="html" keys="IMPALA-2696"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2722" scope="external" format="html" keys="IMPALA-2722"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2728" scope="external" format="html" keys="IMPALA-2728"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2731" scope="external" format="html" keys="IMPALA-2731"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2736" scope="external" format="html" keys="IMPALA-2736"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2742" scope="external" format="html" keys="IMPALA-2742"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2747" scope="external" format="html" keys="IMPALA-2747"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2749" scope="external" format="html" keys="IMPALA-2749"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2767" scope="external" format="html" keys="IMPALA-2767"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2788" scope="external" format="html" keys="IMPALA-2788"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2829" scope="external" format="html" keys="IMPALA-2829"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2835" scope="external" format="html" keys="IMPALA-2835"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2843" scope="external" format="html" keys="IMPALA-2843"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2844" scope="external" format="html" keys="IMPALA-2844"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2867" scope="external" format="html" keys="IMPALA-2867"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-288" scope="external" format="html" keys="IMPALA-288"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-289" scope="external" format="html" keys="IMPALA-289"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2914" scope="external" format="html" keys="IMPALA-2914"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-292" scope="external" format="html" keys="IMPALA-292"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-293" scope="external" format="html" keys="IMPALA-293"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2940" scope="external" format="html" keys="IMPALA-2940"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-295" scope="external" format="html" keys="IMPALA-295"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2956" scope="external" format="html" keys="IMPALA-2956"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-2986" scope="external" format="html" keys="IMPALA-2986"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-30" scope="external" format="html" keys="IMPALA-30"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-300" scope="external" format="html" keys="IMPALA-300"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3001" scope="external" format="html" keys="IMPALA-3001"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3006" scope="external" format="html" keys="IMPALA-3006"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3007" scope="external" format="html" keys="IMPALA-3007"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3008" scope="external" format="html" keys="IMPALA-3008"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3035" scope="external" format="html" keys="IMPALA-3035"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3039" scope="external" format="html" keys="IMPALA-3039"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3044" scope="external" format="html" keys="IMPALA-3044"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3046" scope="external" format="html" keys="IMPALA-3046"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3054" scope="external" format="html" keys="IMPALA-3054"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3056" scope="external" format="html" keys="IMPALA-3056"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3069" scope="external" format="html" keys="IMPALA-3069"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3077" scope="external" format="html" keys="IMPALA-3077"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3082" scope="external" format="html" keys="IMPALA-3082"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3084" scope="external" format="html" keys="IMPALA-3084"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3092" scope="external" format="html" keys="IMPALA-3092"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3094" scope="external" format="html" keys="IMPALA-3094"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3095" scope="external" format="html" keys="IMPALA-3095"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3105" scope="external" format="html" keys="IMPALA-3105"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-312" scope="external" format="html" keys="IMPALA-312"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3126" scope="external" format="html" keys="IMPALA-3126"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3133" scope="external" format="html" keys="IMPALA-3133"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3155" scope="external" format="html" keys="IMPALA-3155"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3163" scope="external" format="html" keys="IMPALA-3163"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3199" scope="external" format="html" keys="IMPALA-3199"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3206" scope="external" format="html" keys="IMPALA-3206"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3210" scope="external" format="html" keys="IMPALA-3210"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3232" scope="external" format="html" keys="IMPALA-3232"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3237" scope="external" format="html" keys="IMPALA-3237"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-325" scope="external" format="html" keys="IMPALA-325"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3269" scope="external" format="html" keys="IMPALA-3269"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3311" scope="external" format="html" keys="IMPALA-3311"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3314" scope="external" format="html" keys="IMPALA-3314"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3317" scope="external" format="html" keys="IMPALA-3317"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-333" scope="external" format="html" keys="IMPALA-333"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3333" scope="external" format="html" keys="IMPALA-3333"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-335" scope="external" format="html" keys="IMPALA-335"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3369" scope="external" format="html" keys="IMPALA-3369"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3375" scope="external" format="html" keys="IMPALA-3375"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3378" scope="external" format="html" keys="IMPALA-3378"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3379" scope="external" format="html" keys="IMPALA-3379"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3385" scope="external" format="html" keys="IMPALA-3385"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3397" scope="external" format="html" keys="IMPALA-3397"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3439" scope="external" format="html" keys="IMPALA-3439"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3441" scope="external" format="html" keys="IMPALA-3441"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3452" scope="external" format="html" keys="IMPALA-3452"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3453" scope="external" format="html" keys="IMPALA-3453"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3471" scope="external" format="html" keys="IMPALA-3471"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3480" scope="external" format="html" keys="IMPALA-3480"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-349" scope="external" format="html" keys="IMPALA-349"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3490" scope="external" format="html" keys="IMPALA-3490"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3494" scope="external" format="html" keys="IMPALA-3494"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3499" scope="external" format="html" keys="IMPALA-3499"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3509" scope="external" format="html" keys="IMPALA-3509"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-351" scope="external" format="html" keys="IMPALA-351"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3554" scope="external" format="html" keys="IMPALA-3554"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3558" scope="external" format="html" keys="IMPALA-3558"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-356" scope="external" format="html" keys="IMPALA-356"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-358" scope="external" format="html" keys="IMPALA-358"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3581" scope="external" format="html" keys="IMPALA-3581"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-364" scope="external" format="html" keys="IMPALA-364"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3662" scope="external" format="html" keys="IMPALA-3662"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3674" scope="external" format="html" keys="IMPALA-3674"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3677" scope="external" format="html" keys="IMPALA-3677"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3679" scope="external" format="html" keys="IMPALA-3679"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3687" scope="external" format="html" keys="IMPALA-3687"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3715" scope="external" format="html" keys="IMPALA-3715"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3716" scope="external" format="html" keys="IMPALA-3716"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3732" scope="external" format="html" keys="IMPALA-3732"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3798" scope="external" format="html" keys="IMPALA-3798"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-38" scope="external" format="html" keys="IMPALA-38"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-3885" scope="external" format="html" keys="IMPALA-3885"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-396" scope="external" format="html" keys="IMPALA-396"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-397" scope="external" format="html" keys="IMPALA-397"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-44" scope="external" format="html" keys="IMPALA-44"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-477" scope="external" format="html" keys="IMPALA-477"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-48" scope="external" format="html" keys="IMPALA-48"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-482" scope="external" format="html" keys="IMPALA-482"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-485" scope="external" format="html" keys="IMPALA-485"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-488" scope="external" format="html" keys="IMPALA-488"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-489" scope="external" format="html" keys="IMPALA-489"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-495" scope="external" format="html" keys="IMPALA-495"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-510" scope="external" format="html" keys="IMPALA-510"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-520" scope="external" format="html" keys="IMPALA-520"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-525" scope="external" format="html" keys="IMPALA-525"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-532" scope="external" format="html" keys="IMPALA-532"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-534" scope="external" format="html" keys="IMPALA-534"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-538" scope="external" format="html" keys="IMPALA-538"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-58" scope="external" format="html" keys="IMPALA-58"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-624" scope="external" format="html" keys="IMPALA-624"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-635" scope="external" format="html" keys="IMPALA-635"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-64" scope="external" format="html" keys="IMPALA-64"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-652" scope="external" format="html" keys="IMPALA-652"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-657" scope="external" format="html" keys="IMPALA-657"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-67" scope="external" format="html" keys="IMPALA-67"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-687" scope="external" format="html" keys="IMPALA-687"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-689" scope="external" format="html" keys="IMPALA-689"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-691" scope="external" format="html" keys="IMPALA-691"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-694" scope="external" format="html" keys="IMPALA-694"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-695" scope="external" format="html" keys="IMPALA-695"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-699" scope="external" format="html" keys="IMPALA-699"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-715" scope="external" format="html" keys="IMPALA-715"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-720" scope="external" format="html" keys="IMPALA-720"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-723" scope="external" format="html" keys="IMPALA-723"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-729" scope="external" format="html" keys="IMPALA-729"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-735" scope="external" format="html" keys="IMPALA-735"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-736" scope="external" format="html" keys="IMPALA-736"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-741" scope="external" format="html" keys="IMPALA-741"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-749" scope="external" format="html" keys="IMPALA-749"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-762" scope="external" format="html" keys="IMPALA-762"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-765" scope="external" format="html" keys="IMPALA-765"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-77" scope="external" format="html" keys="IMPALA-77"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-777" scope="external" format="html" keys="IMPALA-777"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-786" scope="external" format="html" keys="IMPALA-786"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-805" scope="external" format="html" keys="IMPALA-805"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-809" scope="external" format="html" keys="IMPALA-809"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-817" scope="external" format="html" keys="IMPALA-817"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-818" scope="external" format="html" keys="IMPALA-818"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-821" scope="external" format="html" keys="IMPALA-821"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-827" scope="external" format="html" keys="IMPALA-827"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-828" scope="external" format="html" keys="IMPALA-828"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-831" scope="external" format="html" keys="IMPALA-831"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-834" scope="external" format="html" keys="IMPALA-834"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-845" scope="external" format="html" keys="IMPALA-845"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-85" scope="external" format="html" keys="IMPALA-85"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-850" scope="external" format="html" keys="IMPALA-850"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-860" scope="external" format="html" keys="IMPALA-860"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-864" scope="external" format="html" keys="IMPALA-864"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-867" scope="external" format="html" keys="IMPALA-867"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-880" scope="external" format="html" keys="IMPALA-880"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-887" scope="external" format="html" keys="IMPALA-887"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-888" scope="external" format="html" keys="IMPALA-888"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-90" scope="external" format="html" keys="IMPALA-90"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-901" scope="external" format="html" keys="IMPALA-901"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-903" scope="external" format="html" keys="IMPALA-903"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-904" scope="external" format="html" keys="IMPALA-904"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-941" scope="external" format="html" keys="IMPALA-941"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-945" scope="external" format="html" keys="IMPALA-945"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-962" scope="external" format="html" keys="IMPALA-962"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-973" scope="external" format="html" keys="IMPALA-973"/>
+  <keydef href="https://issues.cloudera.org/browse/IMPALA-995" scope="external" format="html" keys="IMPALA-995"/>
+
+  <!-- Pointers to very old internal Cloudera JIRA URLs. Don't work from outside Cloudera. -->
+  <keydef href="https://jira.cloudera.com/browse/CDH-20478" scope="external" format="html" keys="CDH-20478"/>
+  <keydef href="https://jira.cloudera.com/browse/CDH-24040" scope="external" format="html" keys="CDH-24040"/>
+  <keydef href="https://jira.cloudera.com/browse/IMP-1188" scope="external" format="html" keys="IMP-1188"/>
+  <keydef href="https://jira.cloudera.com/browse/IMP-1258" scope="external" format="html" keys="IMP-1258"/>
+  <keydef href="https://jira.cloudera.com/browse/IMP-137" scope="external" format="html" keys="IMP-137"/>
+  <keydef href="https://jira.cloudera.com/browse/IMP-175" scope="external" format="html" keys="IMP-175"/>
+  <keydef href="https://jira.cloudera.com/browse/IMP-231" scope="external" format="html" keys="IMP-231"/>
+  <keydef href="https://jira.cloudera.com/browse/IMP-270" scope="external" format="html" keys="IMP-270"/>
+  <keydef href="https://jira.cloudera.com/browse/IMP-298" scope="external" format="html" keys="IMP-298"/>
+  <keydef href="https://jira.cloudera.com/browse/IMP-469" scope="external" format="html" keys="IMP-469"/>
+  <keydef href="https://jira.cloudera.com/browse/IMP-474" scope="external" format="html" keys="IMP-474"/>
+  <keydef href="https://jira.cloudera.com/browse/IMP-491" scope="external" format="html" keys="IMP-491"/>
+  <keydef href="https://jira.cloudera.com/browse/IMP-497" scope="external" format="html" keys="IMP-497"/>
+  <keydef href="https://jira.cloudera.com/browse/IMP-522" scope="external" format="html" keys="IMP-522"/>
+  <keydef href="https://jira.cloudera.com/browse/IMP-535" scope="external" format="html" keys="IMP-535"/>
+  <keydef href="https://jira.cloudera.com/browse/IMP-56" scope="external" format="html" keys="IMP-56"/>
+  <keydef href="https://jira.cloudera.com/browse/IMP-581" scope="external" format="html" keys="IMP-581"/>
+  <keydef href="https://jira.cloudera.com/browse/IMP-601" scope="external" format="html" keys="IMP-601"/>
+  <keydef href="https://jira.cloudera.com/browse/IMP-611" scope="external" format="html" keys="IMP-611"/>
+  <keydef href="https://jira.cloudera.com/browse/TSB-168" scope="external" format="html" keys="TSB-168"/>
+
+<!-- Short form of mapping from Impala release to vendor-specific releases, for use in headings. -->
+  <keydef keys="impala28"><topicmeta><keywords><keyword>CDH 5.10</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala27"><topicmeta><keywords><keyword>CDH 5.9</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala26"><topicmeta><keywords><keyword>CDH 5.8</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala25"><topicmeta><keywords><keyword>CDH 5.7</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala24"><topicmeta><keywords><keyword>CDH 5.6</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala23"><topicmeta><keywords><keyword>CDH 5.5</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala22"><topicmeta><keywords><keyword>CDH 5.4</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala21"><topicmeta><keywords><keyword>CDH 5.3</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala20"><topicmeta><keywords><keyword>CDH 5.2</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala14"><topicmeta><keywords><keyword>CDH 5.1</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala13"><topicmeta><keywords><keyword>CDH 5.0</keyword></keywords></topicmeta></keydef>
+
+<!-- Short form of mapping from Impala release to vendor-specific releases, for use in running text. -->
+  <keydef keys="impala28_full"><topicmeta><keywords><keyword>CDH 5.10 / Impala 2.8</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala27_full"><topicmeta><keywords><keyword>CDH 5.9 / Impala 2.7</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala26_full"><topicmeta><keywords><keyword>CDH 5.8 / Impala 2.6</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala25_full"><topicmeta><keywords><keyword>CDH 5.7 / Impala 2.5</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala24_full"><topicmeta><keywords><keyword>CDH 5.6 / Impala 2.4</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala23_full"><topicmeta><keywords><keyword>CDH 5.5 / Impala 2.3</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala22_full"><topicmeta><keywords><keyword>CDH 5.4 / Impala 2.2</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala21_full"><topicmeta><keywords><keyword>CDH 5.3 / Impala 2.1</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala20_full"><topicmeta><keywords><keyword>CDH 5.2 / Impala 2.0</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala14_full"><topicmeta><keywords><keyword>CDH 5.1 / Impala 1.4</keyword></keywords></topicmeta></keydef>
+  <keydef keys="impala13_full"><topicmeta><keywords><keyword>CDH 5.0 / Impala 1.3</keyword></keywords></topicmeta></keydef>
+
+<!-- Here is the former site of the Release Notes. Experimenting with moving those to the end for better PDF experience. -->
+<!-- See if there's a way to move include the Release Notes here in HTML, but after Installing/Using for PDF. -->
+<!-- This is the former contents of Installing-and-Using-Impala_xi42979.ditamap. -->
+  <topicref href="topics/impala_intro.xml" audience="standalone"/>
+<!-- Look at how this section works in the 5.2 library, esp. 'components' which AIUI is reused elsewhere. -->
+  <topicref href="topics/impala_concepts.xml">
+  <keydef href="topics/impala_concepts.xml" keys="concepts"/>
+    <topicref href="topics/impala_components.xml"/>
+    <keydef href="topics/impala_components.xml" keys="intro_components"/>
+    <topicref href="topics/impala_development.xml"/>
+    <keydef href="topics/impala_development.xml" keys="intro_dev"/>
+    <topicref href="topics/impala_hadoop.xml"/>
+    <keydef href="topics/impala_hadoop.xml" keys="intro_hadoop"/>
+  </topicref>
+  <topicref href="topics/impala_planning.xml">
+  <keydef href="topics/impala_planning.xml" keys="planning"/>
+    <topicref href="topics/impala_prereqs.xml#prereqs"/>
+    <keydef href="topics/impala_prereqs.xml" keys="prereqs"/>
+    <topicref href="topics/impala_cluster_sizing.xml"/>
+    <keydef href="topics/impala_cluster_sizing.xml" keys="cluster_sizing"/>
+    <topicref href="topics/impala_schema_design.xml"/>
+    <keydef href="topics/impala_schema_design.xml" keys="schema_design"/>
+  </topicref>
+  <topicref audience="standalone" href="topics/impala_install.xml#install">
+  <keydef href="topics/impala_install.xml" keys="install"/>
+    <topicref href="topics/impala_cm_installation.xml#cm_installation"/>
+    <keydef href="topics/impala_cm_installation.xml" keys="cm_installation"/>
+    <topicref href="topics/impala_noncm_installation.xml#noncm_installation"/>
+    <keydef href="topics/impala_noncm_installation.xml" keys="noncm_installation"/>
+<!-- Removed when Hue docs were removed from main library. Might be worth resurrecting someday as general Impala+Hue topic. -->
+<!--		<topicref href="impala_hue_xi20559.xml"/> -->
+  </topicref>
+  <topicref audience="standalone" href="topics/impala_config.xml">
+  <keydef href="topics/impala_config.xml" keys="config"/>
+    <topicref href="topics/impala_config_performance.xml"/>
+    <keydef href="topics/impala_config_performance.xml" keys="config_performance"/>
+    <topicref href="topics/impala_odbc.xml"/>
+    <keydef href="topics/impala_odbc.xml" keys="impala_odbc"/>
+    <topicref href="topics/impala_jdbc.xml"/>
+    <keydef href="topics/impala_jdbc.xml" keys="impala_jdbc"/>
+  </topicref>
+  <topicref audience="standalone" href="topics/impala_upgrading.xml"/>
+  <keydef href="topics/impala_upgrading.xml" keys="upgrading"/>
+  <topicref audience="standalone" href="topics/impala_processes.xml">
+  <keydef href="topics/impala_processes.xml" keys="processes"/>
+    <topicref href="topics/impala_config_options.xml"/>
+    <keydef href="topics/impala_config_options.xml" keys="config_options"/>
+  </topicref>
+  <topicref href="topics/impala_tutorial.xml"/>
+  <keydef href="topics/impala_tutorial.xml" keys="tutorial"/>
+  <topicref href="topics/impala_admin.xml">
+  <keydef href="topics/impala_admin.xml" keys="admin"/>
+    <topicref audience="standalone" href="topics/impala_admission.xml"/>
+    <keydef href="topics/impala_admission.xml" keys="admission_control"/>
+    <topicref audience="standalone" href="topics/impala_resource_management.xml"/>
+    <keydef href="topics/impala_resource_management.xml" keys="resource_management"/>
+    <topicref href="topics/impala_howto_rm.xml"/>
+    <keydef href="topics/impala_howto_rm.xml" keys="howto_impala_rm"/>
+    <topicref href="topics/impala_timeouts.xml"/>
+    <keydef href="topics/impala_timeouts.xml" keys="timeouts"/>
+    <topicref href="topics/impala_proxy.xml"/>
+    <keydef href="topics/impala_proxy.xml" keys="proxy"/>
+    <topicref href="topics/impala_disk_space.xml"/>
+    <keydef href="topics/impala_disk_space.xml" keys="disk_space"/>
+    <topicref audience="integrated" href="topics/impala_auditing.xml"/>
+    <keydef href="topics/impala_auditing.xml" keys="auditing"/>
+    <topicref audience="integrated" href="topics/impala_lineage.xml"/>
+    <keydef href="topics/impala_lineage.xml" keys="lineage"/>
+  </topicref>
+  <topicref audience="standalone" href="topics/impala_security.xml">
+  <keydef href="topics/impala_security.xml" keys="security"/>
+    <topicref href="topics/impala_security_guidelines.xml"/>
+    <keydef href="topics/impala_security_guidelines.xml" keys="security_guidelines"/>
+    <topicref href="topics/impala_security_files.xml"/>
+    <keydef href="topics/impala_security_files.xml" keys="secure_files"/>
+    <topicref href="topics/impala_security_install.xml"/>
+    <keydef href="topics/impala_security_install.xml" keys="security_install"/>
+    <topicref href="topics/impala_security_metastore.xml"/>
+    <keydef href="topics/impala_security_metastore.xml" keys="security_metastore"/>
+    <topicref href="topics/impala_security_webui.xml"/>
+    <keydef href="topics/impala_security_webui.xml" keys="security_webui"/>
+    <topicref href="topics/impala_ssl.xml"/>
+    <keydef href="topics/impala_ssl.xml" keys="ssl"/>
+    <topicref href="topics/impala_authorization.xml"/>
+    <keydef href="topics/impala_authorization.xml" keys="authorization"/>
+    <topicref href="topics/impala_authentication.xml">
+    <keydef href="topics/impala_authentication.xml" keys="authentication"/>
+      <topicref href="topics/impala_kerberos.xml"/>
+      <keydef href="topics/impala_kerberos.xml" keys="kerberos"/>
+      <topicref href="topics/impala_ldap.xml"/>
+      <keydef href="topics/impala_ldap.xml" keys="ldap"/>
+      <topicref href="topics/impala_mixed_security.xml"/>
+      <keydef href="topics/impala_mixed_security.xml" keys="mixed_security"/>
+      <topicref href="topics/impala_delegation.xml"/>
+      <keydef href="topics/impala_delegation.xml" keys="delegation"/>
+    </topicref>
+    <topicref href="topics/impala_auditing.xml"/>
+    <keydef href="topics/impala_auditing.xml" keys="auditing"/>
+    <topicref href="topics/impala_lineage.xml"/>
+    <keydef href="topics/impala_lineage.xml" keys="lineage"/>
+  </topicref>
+  <topicref href="topics/impala_langref.xml">
+  <keydef href="topics/impala_langref.xml" keys="langref"/>
+    <topicref href="topics/impala_comments.xml"/>
+    <keydef href="topics/impala_comments.xml" keys="comments"/>
+    <topicref href="topics/impala_datatypes.xml">
+    <keydef href="topics/impala_datatypes.xml" keys="datatypes"/>
+      <topicref href="topics/impala_array.xml"/>
+      <keydef href="topics/impala_array.xml" keys="array"/>
+      <topicref href="topics/impala_bigint.xml"/>
+      <keydef href="topics/impala_bigint.xml" keys="bigint"/>
+      <topicref href="topics/impala_boolean.xml"/>
+      <keydef href="topics/impala_boolean.xml" keys="boolean"/>
+      <topicref href="topics/impala_char.xml"/>
+      <keydef href="topics/impala_char.xml" keys="char"/>
+      <topicref href="topics/impala_decimal.xml"/>
+      <keydef href="topics/impala_decimal.xml" keys="decimal"/>
+      <topicref audience="Cloudera" href="topics/impala_date.xml"/>
+      <topicref href="topics/impala_double.xml"/>
+      <keydef href="topics/impala_double.xml" keys="double"/>
+      <topicref href="topics/impala_float.xml"/>
+      <keydef href="topics/impala_float.xml" keys="float"/>
+      <topicref href="topics/impala_int.xml"/>
+      <keydef href="topics/impala_int.xml" keys="int"/>
+      <topicref href="topics/impala_map.xml"/>
+      <keydef href="topics/impala_map.xml" keys="map"/>
+      <topicref href="topics/impala_real.xml"/>
+      <keydef href="topics/impala_real.xml" keys="real"/>
+      <topicref href="topics/impala_smallint.xml"/>
+      <keydef href="topics/impala_smallint.xml" keys="smallint"/>
+      <topicref href="topics/impala_string.xml"/>
+      <keydef href="topics/impala_string.xml" keys="string"/>
+      <topicref href="topics/impala_struct.xml"/>
+      <keydef href="topics/impala_struct.xml" keys="struct"/>
+      <topicref href="topics/impala_timestamp.xml"/>
+      <keydef href="topics/impala_timestamp.xml" keys="timestamp"/>
+      <topicref href="topics/impala_tinyint.xml"/>
+      <keydef href="topics/impala_tinyint.xml" keys="tinyint"/>
+      <topicref href="topics/impala_varchar.xml"/>
+      <keydef href="topics/impala_varchar.xml" keys="varchar"/>
+      <topicref href="topics/impala_complex_types.xml"/>
+      <keydef href="topics/impala_complex_types.xml" keys="complex_types"/>
+    </topicref>
+    <topicref href="topics/impala_literals.xml"/>
+    <keydef href="topics/impala_literals.xml" keys="literals"/>
+    <topicref href="topics/impala_operators.xml"/>
+    <keydef href="topics/impala_operators.xml" keys="operators"/>
+    <topicref href="topics/impala_schema_objects.xml">
+    <keydef href="topics/impala_schema_objects.xml" keys="schema_objects"/>
+      <topicref href="topics/impala_aliases.xml"/>
+      <keydef href="topics/impala_aliases.xml" keys="aliases"/>
+      <topicref href="topics/impala_databases.xml"/>
+      <keydef href="topics/impala_databases.xml" keys="databases"/>
+      <topicref audience="Cloudera" href="topics/impala_data_sources.xml"/>
+      <keydef href="topics/impala_data_sources.xml" keys="data_sources"/>
+      <topicref href="topics/impala_functions_overview.xml"/>
+      <keydef href="topics/impala_functions_overview.xml" keys="functions"/>
+      <topicref href="topics/impala_identifiers.xml"/>
+      <keydef href="topics/impala_identifiers.xml" keys="identifiers"/>
+      <topicref href="topics/impala_tables.xml"/>
+      <keydef href="topics/impala_tables.xml" keys="tables"/>
+      <topicref href="topics/impala_views.xml"/>
+      <keydef href="topics/impala_views.xml" keys="views"/>
+    </topicref>
+    <topicref href="topics/impala_langref_sql.xml">
+    <keydef href="topics/impala_langref_sql.xml" keys="langref_sql"/>
+      <topicref href="topics/impala_ddl.xml"/>
+      <keydef href="topics/impala_ddl.xml" keys="ddl"/>
+      <topicref href="topics/impala_dml.xml"/>
+      <keydef href="topics/impala_dml.xml" keys="dml"/>
+      <topicref href="topics/impala_alter_table.xml"/>
+      <keydef href="topics/impala_alter_table.xml" keys="alter_table"/>
+      <topicref href="topics/impala_alter_view.xml"/>
+      <keydef href="topics/impala_alter_view.xml" keys="alter_view"/>
+      <topicref href="topics/impala_compute_stats.xml"/>
+      <keydef href="topics/impala_compute_stats.xml" keys="compute_stats"/>
+      <topicref audience="Cloudera" href="topics/impala_create_data_source.xml"/>
+      <topicref href="topics/impala_create_database.xml"/>
+      <keydef href="topics/impala_create_database.xml" keys="create_database"/>
+      <topicref href="topics/impala_create_function.xml"/>
+      <keydef href="topics/impala_create_function.xml" keys="create_function"/>
+      <topicref href="topics/impala_create_role.xml"/>
+      <keydef href="topics/impala_create_role.xml" keys="create_role"/>
+      <topicref href="topics/impala_create_table.xml"/>
+      <keydef href="topics/impala_create_table.xml" keys="create_table"/>
+      <topicref href="topics/impala_create_view.xml"/>
+      <keydef href="topics/impala_create_view.xml" keys="create_view"/>
+      <topicref href="topics/impala_delete.xml"/>
+      <keydef href="topics/impala_delete.xml" keys="delete"/>
+      <topicref href="topics/impala_describe.xml"/>
+      <keydef href="topics/impala_describe.xml" keys="describe"/>
+      <topicref audience="Cloudera" href="topics/impala_drop_data_source.xml"/>
+      <topicref href="topics/impala_drop_database.xml"/>
+      <keydef href="topics/impala_drop_database.xml" keys="drop_database"/>
+      <topicref href="topics/impala_drop_function.xml"/>
+      <keydef href="topics/impala_drop_function.xml" keys="drop_function"/>
+      <topicref href="topics/impala_drop_role.xml"/>
+      <keydef href="topics/impala_drop_role.xml" keys="drop_role"/>
+      <topicref href="topics/impala_drop_stats.xml"/>
+      <keydef href="topics/impala_drop_stats.xml" keys="drop_stats"/>
+      <topicref href="topics/impala_drop_table.xml"/>
+      <keydef href="topics/impala_drop_table.xml" keys="drop_table"/>
+      <topicref href="topics/impala_drop_view.xml"/>
+      <keydef href="topics/impala_drop_view.xml" keys="drop_view"/>
+      <topicref href="topics/impala_explain.xml"/>
+      <keydef href="topics/impala_explain.xml" keys="explain"/>
+      <topicref href="topics/impala_grant.xml"/>
+      <keydef href="topics/impala_grant.xml" keys="grant"/>
+      <topicref href="topics/impala_insert.xml"/>
+      <keydef href="topics/impala_insert.xml" keys="insert"/>
+      <topicref href="topics/impala_invalidate_metadata.xml"/>
+      <keydef href="topics/impala_invalidate_metadata.xml" keys="invalidate_metadata"/>
+      <topicref href="topics/impala_load_data.xml"/>
+      <keydef href="topics/impala_load_data.xml" keys="load_data"/>
+      <topicref href="topics/impala_refresh.xml"/>
+      <keydef href="topics/impala_refresh.xml" keys="refresh"/>
+      <topicref href="topics/impala_revoke.xml"/>
+      <keydef href="topics/impala_revoke.xml" keys="revoke"/>
+      <topicref href="topics/impala_select.xml">
+      <keydef href="topics/impala_select.xml" keys="select"/>
+        <topicref href="topics/impala_joins.xml"/>
+        <keydef href="topics/impala_joins.xml" keys="joins"/>
+        <topicref href="topics/impala_order_by.xml"/>
+        <keydef href="

<TRUNCATED>


[37/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_create_function.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_create_function.xml b/docs/topics/impala_create_function.xml
new file mode 100644
index 0000000..5b81f50
--- /dev/null
+++ b/docs/topics/impala_create_function.xml
@@ -0,0 +1,492 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.2" id="create_function">
+
+  <title>CREATE FUNCTION Statement</title>
+  <titlealts audience="PDF"><navtitle>CREATE FUNCTION</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="Schemas"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="UDFs"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">CREATE FUNCTION statement</indexterm>
+      Creates a user-defined function (UDF), which you can use to implement custom logic during
+      <codeph>SELECT</codeph> or <codeph>INSERT</codeph> operations.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p>
+      The syntax is different depending on whether you create a scalar UDF, which is called once for each row and
+      implemented by a single function, or a user-defined aggregate function (UDA), which is implemented by
+      multiple functions that compute intermediate results across sets of rows.
+    </p>
+
+    <p rev="2.5.0 IMPALA-2843 CDH-39148">
+      In <keyword keyref="impala25_full"/> and higher, the syntax is also different for creating or dropping scalar Java-based UDFs.
+      The statements for Java UDFs use a new syntax, without any argument types or return type specified. Java-based UDFs
+      created using the new syntax persist across restarts of the Impala catalog server, and can be shared transparently
+      between Impala and Hive.
+    </p>
+
+    <p>
+      To create a persistent scalar C++ UDF with <codeph>CREATE FUNCTION</codeph>:
+    </p>
+
+<codeblock>CREATE FUNCTION [IF NOT EXISTS] [<varname>db_name</varname>.]<varname>function_name</varname>([<varname>arg_type</varname>[, <varname>arg_type</varname>...])
+  RETURNS <varname>return_type</varname>
+  LOCATION '<varname>hdfs_path_to_dot_so</varname>'
+  SYMBOL='<varname>symbol_name</varname>'</codeblock>
+
+    <p rev="2.5.0 IMPALA-2843 CDH-39148">
+      To create a persistent Java UDF with <codeph>CREATE FUNCTION</codeph>:
+<codeblock>CREATE FUNCTION [IF NOT EXISTS] [<varname>db_name</varname>.]<varname>function_name</varname>
+  LOCATION '<varname>hdfs_path_to_jar</varname>'
+  SYMBOL='<varname>class_name</varname>'</codeblock>
+    </p>
+
+<!--
+Examples:
+CREATE FUNCTION IF NOT EXISTS foo location '/path/to/jar' SYMBOL='TestUdf';
+CREATE FUNCTION bar location '/path/to/jar' SYMBOL='TestUdf2';
+DROP FUNCTION foo;
+DROP FUNCTION IF EXISTS bar;
+-->
+
+    <p>
+      To create a persistent UDA, which must be written in C++, issue a <codeph>CREATE AGGREGATE FUNCTION</codeph> statement:
+    </p>
+
+<codeblock>CREATE [AGGREGATE] FUNCTION [IF NOT EXISTS] [<varname>db_name</varname>.]<varname>function_name</varname>([<varname>arg_type</varname>[, <varname>arg_type</varname>...])
+  RETURNS <varname>return_type</varname>
+  LOCATION '<varname>hdfs_path</varname>'
+  [INIT_FN='<varname>function</varname>]
+  UPDATE_FN='<varname>function</varname>
+  MERGE_FN='<varname>function</varname>
+  [PREPARE_FN='<varname>function</varname>]
+  [CLOSEFN='<varname>function</varname>]
+  <ph rev="2.0.0">[SERIALIZE_FN='<varname>function</varname>]</ph>
+  [FINALIZE_FN='<varname>function</varname>]
+  <ph rev="2.3.0 IMPALA-1829 CDH-30572">[INTERMEDIATE <varname>type_spec</varname>]</ph></codeblock>
+
+    <p conref="../shared/impala_common.xml#common/ddl_blurb"/>
+
+    <p>
+      <b>Varargs notation:</b>
+    </p>
+
+    <note rev="CDH-39271 CDH-38572">
+      <p rev="CDH-39271 CDH-38572">
+        Variable-length argument lists are supported for C++ UDFs, but currently not for Java UDFs.
+      </p>
+    </note>
+
+    <p>
+      If the underlying implementation of your function accepts a variable number of arguments:
+    </p>
+
+    <ul>
+      <li>
+        The variable arguments must go last in the argument list.
+      </li>
+
+      <li>
+        The variable arguments must all be of the same type.
+      </li>
+
+      <li>
+        You must include at least one instance of the variable arguments in every function call invoked from SQL.
+      </li>
+
+      <li>
+        You designate the variable portion of the argument list in the <codeph>CREATE FUNCTION</codeph> statement
+        by including <codeph>...</codeph> immediately after the type name of the first variable argument. For
+        example, to create a function that accepts an <codeph>INT</codeph> argument, followed by a
+        <codeph>BOOLEAN</codeph>, followed by one or more <codeph>STRING</codeph> arguments, your <codeph>CREATE
+        FUNCTION</codeph> statement would look like:
+<codeblock>CREATE FUNCTION <varname>func_name</varname> (INT, BOOLEAN, STRING ...)
+  RETURNS <varname>type</varname> LOCATION '<varname>path</varname>' SYMBOL='<varname>entry_point</varname>';
+</codeblock>
+      </li>
+    </ul>
+
+    <p rev="CDH-39271 CDH-38572">
+      See <xref href="impala_udf.xml#udf_varargs"/> for how to code a C++ UDF to accept
+      variable-length argument lists.
+    </p>
+
+    <p>
+      <b>Scalar and aggregate functions:</b>
+    </p>
+
+    <p>
+      The simplest kind of user-defined function returns a single scalar value each time it is called, typically
+      once for each row in the result set. This general kind of function is what is usually meant by UDF.
+      User-defined aggregate functions (UDAs) are a specialized kind of UDF that produce a single value based on
+      the contents of multiple rows. You usually use UDAs in combination with a <codeph>GROUP BY</codeph> clause to
+      condense a large result set into a smaller one, or even a single row summarizing column values across an
+      entire table.
+    </p>
+
+    <p>
+      You create UDAs by using the <codeph>CREATE AGGREGATE FUNCTION</codeph> syntax. The clauses
+      <codeph>INIT_FN</codeph>, <codeph>UPDATE_FN</codeph>, <codeph>MERGE_FN</codeph>,
+      <ph rev="2.0.0"><codeph>SERIALIZE_FN</codeph>,</ph> <codeph>FINALIZE_FN</codeph>, and
+      <codeph>INTERMEDIATE</codeph> only apply when you create a UDA rather than a scalar UDF.
+    </p>
+
+    <p>
+      The <codeph>*_FN</codeph> clauses specify functions to call at different phases of function processing.
+    </p>
+
+    <ul>
+      <li>
+        <b>Initialize:</b> The function you specify with the <codeph>INIT_FN</codeph> clause does any initial
+        setup, such as initializing member variables in internal data structures. This function is often a stub for
+        simple UDAs. You can omit this clause and a default (no-op) function will be used.
+      </li>
+
+      <li>
+        <b>Update:</b> The function you specify with the <codeph>UPDATE_FN</codeph> clause is called once for each
+        row in the original result set, that is, before any <codeph>GROUP BY</codeph> clause is applied. A separate
+        instance of the function is called for each different value returned by the <codeph>GROUP BY</codeph>
+        clause. The final argument passed to this function is a pointer, to which you write an updated value based
+        on its original value and the value of the first argument.
+      </li>
+
+      <li>
+        <b>Merge:</b> The function you specify with the <codeph>MERGE_FN</codeph> clause is called an arbitrary
+        number of times, to combine intermediate values produced by different nodes or different threads as Impala
+        reads and processes data files in parallel. The final argument passed to this function is a pointer, to
+        which you write an updated value based on its original value and the value of the first argument.
+      </li>
+
+      <li rev="2.0.0">
+        <b>Serialize:</b> The function you specify with the <codeph>SERIALIZE_FN</codeph> clause frees memory
+        allocated to intermediate results. It is required if any memory was allocated by the Allocate function in
+        the Init, Update, or Merge functions, or if the intermediate type contains any pointers. See
+        <xref href="https://github.com/cloudera/impala-udf-samples/blob/master/uda-sample.cc" scope="external" format="html">the
+        UDA code samples</xref> for details.
+      </li>
+
+      <li>
+        <b>Finalize:</b> The function you specify with the <codeph>FINALIZE_FN</codeph> clause does any required
+        teardown for resources acquired by your UDF, such as freeing memory, closing file handles if you explicitly
+        opened any files, and so on. This function is often a stub for simple UDAs. You can omit this clause and a
+        default (no-op) function will be used. It is required in UDAs where the final return type is different than
+        the intermediate type. or if any memory was allocated by the Allocate function in the Init, Update, or
+        Merge functions. See
+        <xref href="https://github.com/cloudera/impala-udf-samples/blob/master/uda-sample.cc" scope="external" format="html">the
+        UDA code samples</xref> for details.
+      </li>
+    </ul>
+
+    <p>
+      If you use a consistent naming convention for each of the underlying functions, Impala can automatically
+      determine the names based on the first such clause, so the others are optional.
+    </p>
+
+    <p audience="Cloudera">
+      The <codeph>INTERMEDIATE</codeph> clause specifies the data type of intermediate values passed from the
+      <q>update</q> phase to the <q>merge</q> phase, and from the <q>merge</q> phase to the <q>finalize</q> phase.
+      You can use any of the existing Impala data types, or the special notation
+      <codeph>CHAR(<varname>n</varname>)</codeph> to allocate a scratch area of <varname>n</varname> bytes for the
+      intermediate result. For example, if the different phases of your UDA pass strings to each other but in the
+      end the function returns a <codeph>BIGINT</codeph> value, you would specify <codeph>INTERMEDIATE
+      STRING</codeph>. Likewise, if the different phases of your UDA pass 2 separate <codeph>BIGINT</codeph> values
+      between them (8 bytes each), you would specify <codeph>INTERMEDIATE CHAR(16)</codeph> so that each function
+      could read from and write to a 16-byte buffer.
+    </p>
+
+    <p>
+      For end-to-end examples of UDAs, see <xref href="impala_udf.xml#udfs"/>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/udfs_no_complex_types"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <ul>
+      <li>
+        You can write Impala UDFs in either C++ or Java. C++ UDFs are new to Impala, and are the recommended format
+        for high performance utilizing native code. Java-based UDFs are compatible between Impala and Hive, and are
+        most suited to reusing existing Hive UDFs. (Impala can run Java-based Hive UDFs but not Hive UDAs.)
+      </li>
+
+      <li rev="2.5.0 IMPALA-1748 CDH-38369 IMPALA-2843 CDH-39148">
+        <keyword keyref="impala25_full"/> introduces UDF improvements to persistence for both C++ and Java UDFs,
+        and better compatibility between Impala and Hive for Java UDFs.
+        See <xref href="impala_udf.xml#udfs"/> for details.
+      </li>
+
+      <li>
+        The body of the UDF is represented by a <codeph>.so</codeph> or <codeph>.jar</codeph> file, which you store
+        in HDFS and the <codeph>CREATE FUNCTION</codeph> statement distributes to each Impala node.
+      </li>
+
+      <li>
+        Impala calls the underlying code during SQL statement evaluation, as many times as needed to process all
+        the rows from the result set. All UDFs are assumed to be deterministic, that is, to always return the same
+        result when passed the same argument values. Impala might or might not skip some invocations of a UDF if
+        the result value is already known from a previous call. Therefore, do not rely on the UDF being called a
+        specific number of times, and do not return different result values based on some external factor such as
+        the current time, a random number function, or an external data source that could be updated while an
+        Impala query is in progress.
+      </li>
+
+      <li>
+        The names of the function arguments in the UDF are not significant, only their number, positions, and data
+        types.
+      </li>
+
+      <li>
+        You can overload the same function name by creating multiple versions of the function, each with a
+        different argument signature. For security reasons, you cannot make a UDF with the same name as any
+        built-in function.
+      </li>
+
+      <li>
+        In the UDF code, you represent the function return result as a <codeph>struct</codeph>. This
+        <codeph>struct</codeph> contains 2 fields. The first field is a <codeph>boolean</codeph> representing
+        whether the value is <codeph>NULL</codeph> or not. (When this field is <codeph>true</codeph>, the return
+        value is interpreted as <codeph>NULL</codeph>.) The second field is the same type as the specified function
+        return type, and holds the return value when the function returns something other than
+        <codeph>NULL</codeph>.
+      </li>
+
+      <li>
+        In the UDF code, you represent the function arguments as an initial pointer to a UDF context structure,
+        followed by references to zero or more <codeph>struct</codeph>s, corresponding to each of the arguments.
+        Each <codeph>struct</codeph> has the same 2 fields as with the return value, a <codeph>boolean</codeph>
+        field representing whether the argument is <codeph>NULL</codeph>, and a field of the appropriate type
+        holding any non-<codeph>NULL</codeph> argument value.
+      </li>
+
+      <li>
+        For sample code and build instructions for UDFs,
+        see <xref href="https://github.com/cloudera/impala/tree/master/be/src/udf_samples" scope="external" format="html">the sample UDFs in the Impala github repo</xref>.
+      </li>
+
+      <li>
+        Because the file representing the body of the UDF is stored in HDFS, it is automatically available to all
+        the Impala nodes. You do not need to manually copy any UDF-related files between servers.
+      </li>
+
+      <li>
+        Because Impala currently does not have any <codeph>ALTER FUNCTION</codeph> statement, if you need to rename
+        a function, move it to a different database, or change its signature or other properties, issue a
+        <codeph>DROP FUNCTION</codeph> statement for the original function followed by a <codeph>CREATE
+        FUNCTION</codeph> with the desired properties.
+      </li>
+
+      <li>
+        Because each UDF is associated with a particular database, either issue a <codeph>USE</codeph> statement
+        before doing any <codeph>CREATE FUNCTION</codeph> statements, or specify the name of the function as
+        <codeph><varname>db_name</varname>.<varname>function_name</varname></codeph>.
+      </li>
+    </ul>
+
+    <p conref="../shared/impala_common.xml#common/sync_ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/compatibility_blurb"/>
+
+    <p>
+      Impala can run UDFs that were created through Hive, as long as they refer to Impala-compatible data types
+      (not composite or nested column types). Hive can run Java-based UDFs that were created through Impala, but
+      not Impala UDFs written in C++.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/current_user_caveat"/>
+
+    <p><b>Persistence:</b></p>
+
+    <p conref="../shared/impala_common.xml#common/udf_persistence_restriction"/>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      For additional examples of all kinds of user-defined functions, see <xref href="impala_udf.xml#udfs"/>.
+    </p>
+
+    <p rev="2.5.0 IMPALA-2843 CDH-39148">
+      The following example shows how to take a Java jar file and make all the functions inside one of its classes
+      into UDFs under a single (overloaded) function name in Impala. Each <codeph>CREATE FUNCTION</codeph> or
+      <codeph>DROP FUNCTION</codeph> statement applies to all the overloaded Java functions with the same name.
+      This example uses the signatureless syntax for <codeph>CREATE FUNCTION</codeph> and <codeph>DROP FUNCTION</codeph>,
+      which is available in <keyword keyref="impala25_full"/> and higher.
+    </p>
+    <p rev="2.5.0 IMPALA-2843 CDH-39148">
+      At the start, the jar file is in the local filesystem. Then it is copied into HDFS, so that it is
+      available for Impala to reference through the <codeph>CREATE FUNCTION</codeph> statement and
+      queries that refer to the Impala function name.
+    </p>
+<codeblock rev="2.5.0 IMPALA-2843 CDH-39148">
+$ jar -tvf udf-examples-cdh570.jar
+     0 Mon Feb 22 04:06:50 PST 2016 META-INF/
+   122 Mon Feb 22 04:06:48 PST 2016 META-INF/MANIFEST.MF
+     0 Mon Feb 22 04:06:46 PST 2016 com/
+     0 Mon Feb 22 04:06:46 PST 2016 com/cloudera/
+     0 Mon Feb 22 04:06:46 PST 2016 com/cloudera/impala/
+  2460 Mon Feb 22 04:06:46 PST 2016 com/cloudera/impala/IncompatibleUdfTest.class
+   541 Mon Feb 22 04:06:46 PST 2016 com/cloudera/impala/TestUdfException.class
+  3438 Mon Feb 22 04:06:46 PST 2016 com/cloudera/impala/JavaUdfTest.class
+  5872 Mon Feb 22 04:06:46 PST 2016 com/cloudera/impala/TestUdf.class
+...
+$ hdfs dfs -put udf-examples-cdh570.jar /user/impala/udfs
+$ hdfs dfs -ls /user/impala/udfs
+Found 2 items
+-rw-r--r--   3 jrussell supergroup        853 2015-10-09 14:05 /user/impala/udfs/hello_world.jar
+-rw-r--r--   3 jrussell supergroup       7366 2016-06-08 14:25 /user/impala/udfs/udf-examples-cdh570.jar
+</codeblock>
+    <p rev="2.5.0 IMPALA-2843 CDH-39148">
+      In <cmdname>impala-shell</cmdname>, the <codeph>CREATE FUNCTION</codeph> refers to the HDFS path of the jar file
+      and the fully qualified class name inside the jar. Each of the functions inside the class becomes an
+      Impala function, each one overloaded under the specified Impala function name.
+    </p>
+<codeblock rev="2.5.0 IMPALA-2843 CDH-39148">
+[localhost:21000] > create function testudf location '/user/impala/udfs/udf-examples-cdh570.jar' symbol='com.cloudera.impala.TestUdf';
+[localhost:21000] > show functions;
++-------------+---------------------------------------+-------------+---------------+
+| return type | signature                             | binary type | is persistent |
++-------------+---------------------------------------+-------------+---------------+
+| BIGINT      | testudf(BIGINT)                       | JAVA        | true          |
+| BOOLEAN     | testudf(BOOLEAN)                      | JAVA        | true          |
+| BOOLEAN     | testudf(BOOLEAN, BOOLEAN)             | JAVA        | true          |
+| BOOLEAN     | testudf(BOOLEAN, BOOLEAN, BOOLEAN)    | JAVA        | true          |
+| DOUBLE      | testudf(DOUBLE)                       | JAVA        | true          |
+| DOUBLE      | testudf(DOUBLE, DOUBLE)               | JAVA        | true          |
+| DOUBLE      | testudf(DOUBLE, DOUBLE, DOUBLE)       | JAVA        | true          |
+| FLOAT       | testudf(FLOAT)                        | JAVA        | true          |
+| FLOAT       | testudf(FLOAT, FLOAT)                 | JAVA        | true          |
+| FLOAT       | testudf(FLOAT, FLOAT, FLOAT)          | JAVA        | true          |
+| INT         | testudf(INT)                          | JAVA        | true          |
+| DOUBLE      | testudf(INT, DOUBLE)                  | JAVA        | true          |
+| INT         | testudf(INT, INT)                     | JAVA        | true          |
+| INT         | testudf(INT, INT, INT)                | JAVA        | true          |
+| SMALLINT    | testudf(SMALLINT)                     | JAVA        | true          |
+| SMALLINT    | testudf(SMALLINT, SMALLINT)           | JAVA        | true          |
+| SMALLINT    | testudf(SMALLINT, SMALLINT, SMALLINT) | JAVA        | true          |
+| STRING      | testudf(STRING)                       | JAVA        | true          |
+| STRING      | testudf(STRING, STRING)               | JAVA        | true          |
+| STRING      | testudf(STRING, STRING, STRING)       | JAVA        | true          |
+| TINYINT     | testudf(TINYINT)                      | JAVA        | true          |
++-------------+---------------------------------------+-------------+---------------+
+</codeblock>
+    <p rev="2.5.0 IMPALA-2843 CDH-39148">
+      These are all simple functions that return their single arguments, or
+      sum, concatenate, and so on their multiple arguments. Impala determines which
+      overloaded function to use based on the number and types of the arguments.
+    </p>
+<codeblock rev="2.5.0 IMPALA-2843 CDH-39148">
+insert into bigint_x values (1), (2), (4), (3);
+select testudf(x) from bigint_x;
++-----------------+
+| udfs.testudf(x) |
++-----------------+
+| 1               |
+| 2               |
+| 4               |
+| 3               |
++-----------------+
+
+insert into int_x values (1), (2), (4), (3);
+select testudf(x, x+1, x*x) from int_x;
++-------------------------------+
+| udfs.testudf(x, x + 1, x * x) |
++-------------------------------+
+| 4                             |
+| 9                             |
+| 25                            |
+| 16                            |
++-------------------------------+
+
+select testudf(x) from string_x;
++-----------------+
+| udfs.testudf(x) |
++-----------------+
+| one             |
+| two             |
+| four            |
+| three           |
++-----------------+
+select testudf(x,x) from string_x;
++--------------------+
+| udfs.testudf(x, x) |
++--------------------+
+| oneone             |
+| twotwo             |
+| fourfour           |
+| threethree         |
++--------------------+
+</codeblock>
+
+    <p rev="2.5.0 IMPALA-2843 CDH-39148">
+      The previous example used the same Impala function name as the name of the class.
+      This example shows how the Impala function name is independent of the underlying
+      Java class or function names. A second <codeph>CREATE FUNCTION</codeph> statement
+      results in a set of overloaded functions all named <codeph>my_func</codeph>,
+      to go along with the overloaded functions all named <codeph>testudf</codeph>.
+    </p>
+<codeblock rev="2.5.0 IMPALA-2843 CDH-39148">
+create function my_func location '/user/impala/udfs/udf-examples-cdh570.jar'
+  symbol='com.cloudera.impala.TestUdf';
+
+show functions;
++-------------+---------------------------------------+-------------+---------------+
+| return type | signature                             | binary type | is persistent |
++-------------+---------------------------------------+-------------+---------------+
+| BIGINT      | my_func(BIGINT)                       | JAVA        | true          |
+| BOOLEAN     | my_func(BOOLEAN)                      | JAVA        | true          |
+| BOOLEAN     | my_func(BOOLEAN, BOOLEAN)             | JAVA        | true          |
+...
+| BIGINT      | testudf(BIGINT)                       | JAVA        | true          |
+| BOOLEAN     | testudf(BOOLEAN)                      | JAVA        | true          |
+| BOOLEAN     | testudf(BOOLEAN, BOOLEAN)             | JAVA        | true          |
+...
+</codeblock>
+    <p rev="2.5.0 IMPALA-2843 CDH-39148">
+      The corresponding <codeph>DROP FUNCTION</codeph> statement with no signature
+      drops all the overloaded functions with that name.
+    </p>
+<codeblock rev="2.5.0 IMPALA-2843 CDH-39148">
+drop function my_func;
+show functions;
++-------------+---------------------------------------+-------------+---------------+
+| return type | signature                             | binary type | is persistent |
++-------------+---------------------------------------+-------------+---------------+
+| BIGINT      | testudf(BIGINT)                       | JAVA        | true          |
+| BOOLEAN     | testudf(BOOLEAN)                      | JAVA        | true          |
+| BOOLEAN     | testudf(BOOLEAN, BOOLEAN)             | JAVA        | true          |
+...
+</codeblock>
+    <p rev="2.5.0 IMPALA-2843 CDH-39148">
+      The signatureless <codeph>CREATE FUNCTION</codeph> syntax for Java UDFs ensures that
+      the functions shown in this example remain available after the Impala service
+      (specifically, the Catalog Server) are restarted.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_udf.xml#udfs"/> for more background information, usage instructions, and examples for
+      Impala UDFs; <xref href="impala_drop_function.xml#drop_function"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_create_role.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_create_role.xml b/docs/topics/impala_create_role.xml
new file mode 100644
index 0000000..2308490
--- /dev/null
+++ b/docs/topics/impala_create_role.xml
@@ -0,0 +1,70 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.4.0" id="create_role">
+
+  <title>CREATE ROLE Statement (<keyword keyref="impala20"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>CREATE ROLE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Sentry"/>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="Roles"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+      <!-- Consider whether to go deeper into categories like Security for the Sentry-related statements. -->
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">CREATE ROLE statement</indexterm>
+<!-- Copied from Sentry docs. Turn into conref. -->
+      The <codeph>CREATE ROLE</codeph> statement creates a role to which privileges can be granted. Privileges can
+      be granted to roles, which can then be assigned to users. A user that has been assigned a role will only be
+      able to exercise the privileges of that role. Only users that have administrative privileges can create/drop
+      roles. By default, the <codeph>hive</codeph>, <codeph>impala</codeph> and <codeph>hue</codeph> users have
+      administrative privileges in Sentry.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>CREATE ROLE <varname>role_name</varname>
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/privileges_blurb"/>
+
+    <p>
+      Only administrative users (those with <codeph>ALL</codeph> privileges on the server, defined in the Sentry
+      policy file) can use this statement.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/compatibility_blurb"/>
+
+    <p>
+      Impala makes use of any roles and privileges specified by the <codeph>GRANT</codeph> and
+      <codeph>REVOKE</codeph> statements in Hive, and Hive makes use of any roles and privileges specified by the
+      <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements in Impala. The Impala <codeph>GRANT</codeph>
+      and <codeph>REVOKE</codeph> statements for privileges do not require the <codeph>ROLE</codeph> keyword to be
+      repeated before each role name, unlike the equivalent Hive statements.
+    </p>
+
+<!-- To do: nail down the new SHOW syntax, e.g. SHOW ROLES, SHOW CURRENT ROLES, SHOW GROUPS. -->
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_authorization.xml#authorization"/>, <xref href="impala_grant.xml#grant"/>,
+      <xref href="impala_revoke.xml#revoke"/>, <xref href="impala_drop_role.xml#drop_role"/>,
+      <xref href="impala_show.xml#show"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_create_table.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_create_table.xml b/docs/topics/impala_create_table.xml
new file mode 100644
index 0000000..ba83fc5
--- /dev/null
+++ b/docs/topics/impala_create_table.xml
@@ -0,0 +1,832 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="create_table" outputclass="impala sql_statement">
+
+  <title outputclass="impala_title sql_statement_title">CREATE TABLE Statement</title>
+  <titlealts audience="PDF"><navtitle>CREATE TABLE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="HDFS Caching"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Schemas"/>
+      <data name="Category" value="S3"/>
+      <!-- <data name="Category" value="Kudu"/> -->
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">CREATE TABLE statement</indexterm>
+      Creates a new table and specifies its characteristics. While creating a table, you optionally specify aspects
+      such as:
+    </p>
+
+    <ul>
+      <li>
+        Whether the table is internal or external.
+      </li>
+
+      <li>
+        The columns and associated data types.
+      </li>
+
+      <li>
+        The columns used for physically partitioning the data.
+      </li>
+
+      <li>
+        The file format for data files.
+      </li>
+
+      <li>
+        The HDFS directory where the data files are located.
+      </li>
+    </ul>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p>
+      The general syntax for creating a table and specifying its columns is as follows:
+    </p>
+
+    <p>
+      <b>Explicit column definitions:</b>
+    </p>
+
+<codeblock>CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [<varname>db_name</varname>.]<varname>table_name</varname>
+  (<varname>col_name</varname> <varname>data_type</varname> [COMMENT '<varname>col_comment</varname>'], ...)
+  [PARTITIONED BY (<varname>col_name</varname> <varname>data_type</varname> [COMMENT '<varname>col_comment</varname>'], ...)]
+  [COMMENT '<varname>table_comment</varname>']
+  [WITH SERDEPROPERTIES ('<varname>key1</varname>'='<varname>value1</varname>', '<varname>key2</varname>'='<varname>value2</varname>', ...)]
+  [
+   [ROW FORMAT <varname>row_format</varname>] [STORED AS <varname>file_format</varname>]
+  ]
+  [LOCATION '<varname>hdfs_path</varname>']
+  [TBLPROPERTIES ('<varname>key1</varname>'='<varname>value1</varname>', '<varname>key2</varname>'='<varname>value2</varname>', ...)]
+<ph rev="1.4.0">  [CACHED IN '<varname>pool_name</varname>'</ph> <ph rev="2.2.0">[WITH REPLICATION = <varname>integer</varname>]</ph> | UNCACHED]
+</codeblock>
+
+    <p>
+      <b>Column definitions inferred from data file:</b>
+    </p>
+
+<codeblock>CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [<varname>db_name</varname>.]<varname>table_name</varname>
+  LIKE PARQUET '<varname>hdfs_path_of_parquet_file</varname>'
+  [COMMENT '<varname>table_comment</varname>']
+  [PARTITIONED BY (<varname>col_name</varname> <varname>data_type</varname> [COMMENT '<varname>col_comment</varname>'], ...)]
+  [WITH SERDEPROPERTIES ('<varname>key1</varname>'='<varname>value1</varname>', '<varname>key2</varname>'='<varname>value2</varname>', ...)]
+  [
+   [ROW FORMAT <varname>row_format</varname>] [STORED AS <varname>file_format</varname>]
+  ]
+  [LOCATION '<varname>hdfs_path</varname>']
+  [TBLPROPERTIES ('<varname>key1</varname>'='<varname>value1</varname>', '<varname>key2</varname>'='<varname>value2</varname>', ...)]
+<ph rev="1.4.0">  [CACHED IN '<varname>pool_name</varname>'</ph> <ph rev="2.2.0">[WITH REPLICATION = <varname>integer</varname>]</ph> | UNCACHED]
+data_type:
+    <varname>primitive_type</varname>
+  | array_type
+  | map_type
+  | struct_type
+</codeblock>
+
+    <p>
+      <b>CREATE TABLE AS SELECT:</b>
+    </p>
+
+<codeblock>CREATE [EXTERNAL] TABLE [IF NOT EXISTS] <varname>db_name</varname>.]<varname>table_name</varname>
+  <ph rev="2.5.0">[PARTITIONED BY (<varname>col_name</varname>[, ...])]</ph>
+  [COMMENT '<varname>table_comment</varname>']
+  [WITH SERDEPROPERTIES ('<varname>key1</varname>'='<varname>value1</varname>', '<varname>key2</varname>'='<varname>value2</varname>', ...)]
+  [
+   [ROW FORMAT <varname>row_format</varname>] <ph rev="CDH-41501">[STORED AS <varname>ctas_file_format</varname>]</ph>
+  ]
+  [LOCATION '<varname>hdfs_path</varname>']
+  [TBLPROPERTIES ('<varname>key1</varname>'='<varname>value1</varname>', '<varname>key2</varname>'='<varname>value2</varname>', ...)]
+<ph rev="1.4.0">  [CACHED IN '<varname>pool_name</varname>'</ph> <ph rev="2.2.0">[WITH REPLICATION = <varname>integer</varname>]</ph> | UNCACHED]
+AS
+  <varname>select_statement</varname></codeblock>
+
+<codeblock>primitive_type:
+    TINYINT
+  | SMALLINT
+  | INT
+  | BIGINT
+  | BOOLEAN
+  | FLOAT
+  | DOUBLE
+  <ph rev="1.4.0">| DECIMAL</ph>
+  | STRING
+  <ph rev="2.0.0">| CHAR</ph>
+  <ph rev="2.0.0">| VARCHAR</ph>
+  | TIMESTAMP
+
+<ph rev="2.3.0">complex_type:
+    struct_type
+  | array_type
+  | map_type
+
+struct_type: STRUCT &lt; <varname>name</varname> : <varname>primitive_or_complex_type</varname> [COMMENT '<varname>comment_string</varname>'], ... &gt;
+
+array_type: ARRAY &lt; <varname>primitive_or_complex_type</varname> &gt;
+
+map_type: MAP &lt; <varname>primitive_type</varname>, <varname>primitive_or_complex_type</varname> &gt;
+</ph>
+row_format:
+  DELIMITED [FIELDS TERMINATED BY '<varname>char</varname>' [ESCAPED BY '<varname>char</varname>']]
+  [LINES TERMINATED BY '<varname>char</varname>']
+
+file_format:
+    PARQUET
+  | TEXTFILE
+  | AVRO
+  | SEQUENCEFILE
+  | RCFILE
+
+<ph rev="CDH-41501">ctas_file_format:
+    PARQUET
+  | TEXTFILE</ph>
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/ddl_blurb"/>
+
+    <!-- Should really have some info up front about all the data types and file formats.
+         Consider adding here, or at least making inline links to the relevant keywords
+         in the syntax spec above. -->
+
+    <p>
+      <b>Column definitions:</b>
+    </p>
+
+    <p>
+      Depending on the form of the <codeph>CREATE TABLE</codeph> statement, the column definitions are
+      required or not allowed.
+    </p>
+
+    <p>
+      With the <codeph>CREATE TABLE AS SELECT</codeph> and <codeph>CREATE TABLE LIKE</codeph>
+      syntax, you do not specify the columns at all; the column names and types are derived from the source table, query,
+      or data file.
+    </p>
+
+    <p>
+      With the basic <codeph>CREATE TABLE</codeph> syntax, you must list one or more columns,
+      its name, type, and optionally a comment, in addition to any columns used as partitioning keys.
+      There is one exception where the column list is not required: when creating an Avro table with the
+      <codeph>STORED AS AVRO</codeph> clause, you can omit the list of columns and specify the same metadata
+      as part of the <codeph>TBLPROPERTIES</codeph> clause.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+    <p rev="2.3.0">
+      The Impala complex types (<codeph>STRUCT</codeph>, <codeph>ARRAY</codeph>, or <codeph>MAP</codeph>)
+      are available in <keyword keyref="impala23_full"/> and higher.
+      Because you can nest these types (for example, to make an array of maps or a struct
+      with an array field), these types are also sometimes referred to as nested types.
+      See <xref href="impala_complex_types.xml#complex_types"/> for usage details.
+    </p>
+
+    <!-- This is kind of an obscure and rare usage scenario. Consider moving all the complex type stuff further down
+         after some of the more common clauses. -->
+    <p rev="2.3.0">
+      Impala can create tables containing complex type columns, with any supported file format.
+      Because currently Impala can only query complex type columns in Parquet tables, creating
+      tables with complex type columns and other file formats such as text is of limited use.
+      For example, you might create a text table including some columns with complex types with Impala, and use Hive
+      as part of your to ingest the nested type data and copy it to an identical Parquet table.
+      Or you might create a partitioned table containing complex type columns using one file format, and
+      use <codeph>ALTER TABLE</codeph> to change the file format of individual partitions to Parquet; Impala
+      can then query only the Parquet-format partitions in that table.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_partitioning"/>
+
+    <p>
+      <b>Internal and external tables (EXTERNAL and LOCATION clauses):</b>
+    </p>
+
+    <p>
+      By default, Impala creates an <q>internal</q> table, where Impala manages the underlying data files for the
+      table, and physically deletes the data files when you drop the table. If you specify the
+      <codeph>EXTERNAL</codeph> clause, Impala treats the table as an <q>external</q> table, where the data files
+      are typically produced outside Impala and queried from their original locations in HDFS, and Impala leaves
+      the data files in place when you drop the table. For details about internal and external tables, see
+      <xref href="impala_tables.xml#tables"/>.
+    </p>
+
+    <p>
+      Typically, for an external table you include a <codeph>LOCATION</codeph> clause to specify the path to the
+      HDFS directory where Impala reads and writes files for the table. For example, if your data pipeline produces
+      Parquet files in the HDFS directory <filepath>/user/etl/destination</filepath>, you might create an external
+      table as follows:
+    </p>
+
+<codeblock>CREATE EXTERNAL TABLE external_parquet (c1 INT, c2 STRING, c3 TIMESTAMP)
+  STORED AS PARQUET LOCATION '/user/etl/destination';
+</codeblock>
+
+    <p>
+      Although the <codeph>EXTERNAL</codeph> and <codeph>LOCATION</codeph> clauses are often specified together,
+      <codeph>LOCATION</codeph> is optional for external tables, and you can also specify <codeph>LOCATION</codeph>
+      for internal tables. The difference is all about whether Impala <q>takes control</q> of the underlying data
+      files and moves them when you rename the table, or deletes them when you drop the table. For more about
+      internal and external tables and how they interact with the <codeph>LOCATION</codeph> attribute, see
+      <xref href="impala_tables.xml#tables"/>.
+    </p>
+
+    <p>
+      <b>Partitioned tables (PARTITIONED BY clause):</b>
+    </p>
+
+    <p>
+      The <codeph>PARTITIONED BY</codeph> clause divides the data files based on the values from one or more
+      specified columns. Impala queries can use the partition metadata to minimize the amount of data that is read
+      from disk or transmitted across the network, particularly during join queries. For details about
+      partitioning, see <xref href="impala_partitioning.xml#partitioning"/>.
+    </p>
+
+    <p rev="2.5.0">
+      Prior to <keyword keyref="impala25_full"/> you could use a partitioned table
+      as the source and copy data from it, but could not specify any partitioning clauses for the new table.
+      In <keyword keyref="impala25_full"/> and higher, you can now use the <codeph>PARTITIONED BY</codeph> clause with a
+      <codeph>CREATE TABLE AS SELECT</codeph> statement. See the examples under the following discussion of
+      the <codeph>CREATE TABLE AS SELECT</codeph> syntax variation.
+    </p>
+
+<!--
+    <p rev="kudu">
+      <b>Partitioning for Kudu tables (DISTRIBUTE BY clause)</b>
+    </p>
+
+    <p rev="kudu">
+      For Kudu tables, you specify logical partitioning across one or more columns using the
+      <codeph>DISTRIBUTE BY</codeph> clause. In contrast to partitioning for HDFS-based tables,
+      multiple values for a partition key column can be located in the same partition.
+      The optional <codeph>HASH</codeph> clause lets you divide one or a set of partition key columns
+      into a specified number of buckets; you can use more than one <codeph>HASH</codeph>
+      clause, specifying a distinct set of partition key columns for each.
+      The optional <codeph>RANGE</codeph> clause further subdivides the partitions, based on
+      a set of literal values for the partition key columns.
+    </p>
+-->
+
+    <p>
+      <b>Specifying file format (STORED AS and ROW FORMAT clauses):</b>
+    </p>
+
+    <p rev="DOCS-1523">
+      The <codeph>STORED AS</codeph> clause identifies the format of the underlying data files. Currently, Impala
+      can query more types of file formats than it can create or insert into. Use Hive to perform any create or
+      data load operations that are not currently available in Impala. For example, Impala can create an Avro,
+      SequenceFile, or RCFile table but cannot insert data into it. There are also Impala-specific procedures for using
+      compression with each kind of file format. For details about working with data files of various formats, see
+      <xref href="impala_file_formats.xml#file_formats"/>.
+    </p>
+
+    <note>
+      In Impala 1.4.0 and higher, Impala can create Avro tables, which formerly required doing the <codeph>CREATE
+      TABLE</codeph> statement in Hive. See <xref href="impala_avro.xml#avro"/> for details and examples.
+    </note>
+
+    <p>
+      By default (when no <codeph>STORED AS</codeph> clause is specified), data files in Impala tables are created
+      as text files with Ctrl-A (hex 01) characters as the delimiter.
+<!-- Verify if ROW FORMAT is entirely ignored outside of text tables, or does it apply somehow to SequenceFile and/or RCFile too? -->
+      Specify the <codeph>ROW FORMAT DELIMITED</codeph> clause to produce or ingest data files that use a different
+      delimiter character such as tab or <codeph>|</codeph>, or a different line end character such as carriage
+      return or newline. When specifying delimiter and line end characters with the <codeph>FIELDS TERMINATED
+      BY</codeph> and <codeph>LINES TERMINATED BY</codeph> clauses, use <codeph>'\t'</codeph> for tab,
+      <codeph>'\n'</codeph> for newline or linefeed, <codeph>'\r'</codeph> for carriage return, and
+      <codeph>\</codeph><codeph>0</codeph> for ASCII <codeph>nul</codeph> (hex 00). For more examples of text
+      tables, see <xref href="impala_txtfile.xml#txtfile"/>.
+    </p>
+
+    <p>
+      The <codeph>ESCAPED BY</codeph> clause applies both to text files that you create through an
+      <codeph>INSERT</codeph> statement to an Impala <codeph>TEXTFILE</codeph> table, and to existing data files
+      that you put into an Impala table directory. (You can ingest existing data files either by creating the table
+      with <codeph>CREATE EXTERNAL TABLE ... LOCATION</codeph>, the <codeph>LOAD DATA</codeph> statement, or
+      through an HDFS operation such as <codeph>hdfs dfs -put <varname>file</varname>
+      <varname>hdfs_path</varname></codeph>.) Choose an escape character that is not used anywhere else in the
+      file, and put it in front of each instance of the delimiter character that occurs within a field value.
+      Surrounding field values with quotation marks does not help Impala to parse fields with embedded delimiter
+      characters; the quotation marks are considered to be part of the column value. If you want to use
+      <codeph>\</codeph> as the escape character, specify the clause in <cmdname>impala-shell</cmdname> as
+      <codeph>ESCAPED BY '\\'</codeph>.
+    </p>
+
+    <note conref="../shared/impala_common.xml#common/thorn"/>
+
+    <p>
+      <b>Cloning tables (LIKE clause):</b>
+    </p>
+
+    <p>
+      To create an empty table with the same columns, comments, and other attributes as another table, use the
+      following variation. The <codeph>CREATE TABLE ... LIKE</codeph> form allows a restricted set of clauses,
+      currently only the <codeph>LOCATION</codeph>, <codeph>COMMENT</codeph>, and <codeph>STORED AS</codeph>
+      clauses.
+    </p>
+
+<codeblock>CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [<varname>db_name</varname>.]<varname>table_name</varname>
+  <ph rev="1.4.0">LIKE { [<varname>db_name</varname>.]<varname>table_name</varname> | PARQUET '<varname>hdfs_path_of_parquet_file</varname>' }</ph>
+  [COMMENT '<varname>table_comment</varname>']
+  [STORED AS <varname>file_format</varname>]
+  [LOCATION '<varname>hdfs_path</varname>']</codeblock>
+
+    <note rev="1.2.0">
+      <p rev="1.2.0">
+        To clone the structure of a table and transfer data into it in a single operation, use the <codeph>CREATE
+        TABLE AS SELECT</codeph> syntax described in the next subsection.
+      </p>
+    </note>
+
+    <p>
+      When you clone the structure of an existing table using the <codeph>CREATE TABLE ... LIKE</codeph> syntax,
+      the new table keeps the same file format as the original one, so you only need to specify the <codeph>STORED
+      AS</codeph> clause if you want to use a different file format, or when specifying a view as the original
+      table. (Creating a table <q>like</q> a view produces a text table by default.)
+    </p>
+
+    <p>
+      Although normally Impala cannot create an HBase table directly, Impala can clone the structure of an existing
+      HBase table with the <codeph>CREATE TABLE ... LIKE</codeph> syntax, preserving the file format and metadata
+      from the original table.
+    </p>
+
+    <p>
+      There are some exceptions to the ability to use <codeph>CREATE TABLE ... LIKE</codeph> with an Avro table.
+      For example, you cannot use this technique for an Avro table that is specified with an Avro schema but no
+      columns. When in doubt, check if a <codeph>CREATE TABLE ... LIKE</codeph> operation works in Hive; if not, it
+      typically will not work in Impala either.
+    </p>
+
+    <p>
+      If the original table is partitioned, the new table inherits the same partition key columns. Because the new
+      table is initially empty, it does not inherit the actual partitions that exist in the original one. To create
+      partitions in the new table, insert data or issue <codeph>ALTER TABLE ... ADD PARTITION</codeph> statements.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/create_table_like_view"/>
+
+    <p>
+      Because <codeph>CREATE TABLE ... LIKE</codeph> only manipulates table metadata, not the physical data of the
+      table, issue <codeph>INSERT INTO TABLE</codeph> statements afterward to copy any data from the original table
+      into the new one, optionally converting the data to a new file format. (For some file formats, Impala can do
+      a <codeph>CREATE TABLE ... LIKE</codeph> to create the table, but Impala cannot insert data in that file
+      format; in these cases, you must load the data in Hive. See
+      <xref href="impala_file_formats.xml#file_formats"/> for details.)
+    </p>
+
+    <p rev="1.2" id="ctas">
+      <b>CREATE TABLE AS SELECT:</b>
+    </p>
+
+    <p>
+      The <codeph>CREATE TABLE AS SELECT</codeph> syntax is a shorthand notation to create a table based on column
+      definitions from another table, and copy data from the source table to the destination table without issuing
+      any separate <codeph>INSERT</codeph> statement. This idiom is so popular that it has its own acronym,
+      <q>CTAS</q>.
+    </p>
+
+    <p>
+      The following examples show how to copy data from a source table <codeph>T1</codeph>
+      to a variety of destinations tables, applying various transformations to the table
+      properties, table layout, or the data itself as part of the operation:
+    </p>
+
+<codeblock>
+-- Sample table to be the source of CTAS operations.
+CREATE TABLE t1 (x INT, y STRING);
+INSERT INTO t1 VALUES (1, 'one'), (2, 'two'), (3, 'three');
+
+-- Clone all the columns and data from one table to another.
+CREATE TABLE clone_of_t1 AS SELECT * FROM t1;
++-------------------+
+| summary           |
++-------------------+
+| Inserted 3 row(s) |
++-------------------+
+
+-- Clone the columns and data, and convert the data to a different file format.
+CREATE TABLE parquet_version_of_t1 STORED AS PARQUET AS SELECT * FROM t1;
++-------------------+
+| summary           |
++-------------------+
+| Inserted 3 row(s) |
++-------------------+
+
+-- Copy only some rows to the new table.
+CREATE TABLE subset_of_t1 AS SELECT * FROM t1 WHERE x >= 2;
++-------------------+
+| summary           |
++-------------------+
+| Inserted 2 row(s) |
++-------------------+
+
+-- Same idea as CREATE TABLE LIKE: clone table layout but do not copy any data.
+CREATE TABLE empty_clone_of_t1 AS SELECT * FROM t1 WHERE 1=0;
++-------------------+
+| summary           |
++-------------------+
+| Inserted 0 row(s) |
++-------------------+
+
+-- Reorder and rename columns and transform the data.
+CREATE TABLE t5 AS SELECT upper(y) AS s, x+1 AS a, 'Entirely new column' AS n FROM t1;
++-------------------+
+| summary           |
++-------------------+
+| Inserted 3 row(s) |
++-------------------+
+SELECT * FROM t5;
++-------+---+---------------------+
+| s     | a | n                   |
++-------+---+---------------------+
+| ONE   | 2 | Entirely new column |
+| TWO   | 3 | Entirely new column |
+| THREE | 4 | Entirely new column |
++-------+---+---------------------+
+</codeblock>
+
+<!-- These are a little heavyweight to get into here. Therefore commenting out.
+     Some overlap with the new column-changing examples in the code listing above.
+Create tables with different column order, names, or types than the original.
+CREATE TABLE some_columns_from_t1 AS SELECT c1, c3, c5 FROM t1;
+CREATE TABLE reordered_columns_from_t1 AS SELECT c4, c3, c1, c2 FROM t1;
+CREATE TABLE synthesized_columns AS SELECT upper(c1) AS all_caps, c2+c3 AS total, "California" AS state FROM t1;</codeblock>
+-->
+
+<!-- CREATE TABLE AS <select> now incorporated up higher in the original syntax diagram. -->
+
+    <p rev="1.2">
+      See <xref href="impala_select.xml#select"/> for details about query syntax for the <codeph>SELECT</codeph>
+      portion of a <codeph>CREATE TABLE AS SELECT</codeph> statement.
+    </p>
+
+    <p rev="1.2">
+      The newly created table inherits the column names that you select from the original table, which you can
+      override by specifying column aliases in the query. Any column or table comments from the original table are
+      not carried over to the new table.
+    </p>
+
+    <note rev="DOCS-1523">
+      When using the <codeph>STORED AS</codeph> clause with a <codeph>CREATE TABLE AS SELECT</codeph>
+      statement, the destination table must be a file format that Impala can write to: currently,
+      text or Parquet. You cannot specify an Avro, SequenceFile, or RCFile table as the destination
+      table for a CTAS operation.
+    </note>
+
+    <p rev="2.5.0">
+      Prior to <keyword keyref="impala25_full"/> you could use a partitioned table
+      as the source and copy data from it, but could not specify any partitioning clauses for the new table.
+      In <keyword keyref="impala25_full"/> and higher, you can now use the <codeph>PARTITIONED BY</codeph> clause with a
+      <codeph>CREATE TABLE AS SELECT</codeph> statement. The following example demonstrates how you can copy
+      data from an unpartitioned table in a <codeph>CREATE TABLE AS SELECT</codeph> operation, creating a new
+      partitioned table in the process. The main syntax consideration is the column order in the <codeph>PARTITIONED BY</codeph>
+      clause and the select list: the partition key columns must be listed last in the select list, in the same
+      order as in the <codeph>PARTITIONED BY</codeph> clause. Therefore, in this case, the column order in the
+      destination table is different from the source table. You also only specify the column names in the
+      <codeph>PARTITIONED BY</codeph> clause, not the data types or column comments.
+    </p>
+
+<codeblock rev="2.5.0">
+create table partitions_no (year smallint, month tinyint, s string);
+insert into partitions_no values (2016, 1, 'January 2016'),
+  (2016, 2, 'February 2016'), (2016, 3, 'March 2016');
+
+-- Prove that the source table is not partitioned.
+show partitions partitions_no;
+ERROR: AnalysisException: Table is not partitioned: ctas_partition_by.partitions_no
+
+-- Create new table with partitions based on column values from source table.
+<b>create table partitions_yes partitioned by (year, month)
+  as select s, year, month from partitions_no;</b>
++-------------------+
+| summary           |
++-------------------+
+| Inserted 3 row(s) |
++-------------------+
+
+-- Prove that the destination table is partitioned.
+show partitions partitions_yes;
++-------+-------+-------+--------+------+...
+| year  | month | #Rows | #Files | Size |...
++-------+-------+-------+--------+------+...
+| 2016  | 1     | -1    | 1      | 13B  |...
+| 2016  | 2     | -1    | 1      | 14B  |...
+| 2016  | 3     | -1    | 1      | 11B  |...
+| Total |       | -1    | 3      | 38B  |...
++-------+-------+-------+--------+------+...
+</codeblock>
+
+  <p rev="2.5.0">
+    The most convenient layout for partitioned tables is with all the
+    partition key columns at the end. The CTAS <codeph>PARTITIONED BY</codeph> syntax
+    requires that column order in the select list, resulting in that same
+    column order in the destination table.
+  </p>
+
+<codeblock rev="2.5.0">
+describe partitions_no;
++-------+----------+---------+
+| name  | type     | comment |
++-------+----------+---------+
+| year  | smallint |         |
+| month | tinyint  |         |
+| s     | string   |         |
++-------+----------+---------+
+
+-- The CTAS operation forced us to put the partition key columns last.
+-- Having those columns last works better with idioms such as SELECT *
+-- for partitioned tables.
+describe partitions_yes;
++-------+----------+---------+
+| name  | type     | comment |
++-------+----------+---------+
+| s     | string   |         |
+| year  | smallint |         |
+| month | tinyint  |         |
++-------+----------+---------+
+</codeblock>
+
+  <p rev="2.5.0">
+    Attempting to use a select list with the partition key columns
+    not at the end results in an error due to a column name mismatch:
+  </p>
+
+<codeblock rev="2.5.0">
+-- We expect this CTAS to fail because non-key column S
+-- comes after key columns YEAR and MONTH in the select list.
+create table partitions_maybe partitioned by (year, month)
+  as select year, month, s from partitions_no;
+ERROR: AnalysisException: Partition column name mismatch: year != month
+</codeblock>
+
+    <p rev="1.2">
+      For example, the following statements show how you can clone all the data in a table, or a subset of the
+      columns and/or rows, or reorder columns, rename them, or construct them out of expressions:
+    </p>
+
+    <p rev="1.2">
+      As part of a CTAS operation, you can convert the data to any file format that Impala can write (currently,
+      <codeph>TEXTFILE</codeph> and <codeph>PARQUET</codeph>). You cannot specify the lower-level properties of a
+      text table, such as the delimiter.
+    </p>
+
+    <p rev="obwl" conref="../shared/impala_common.xml#common/insert_sort_blurb"/>
+
+    <p rev="1.4.0">
+      <b>CREATE TABLE LIKE PARQUET:</b>
+    </p>
+
+    <p rev="1.4.0">
+      The variation <codeph>CREATE TABLE ... LIKE PARQUET '<varname>hdfs_path_of_parquet_file</varname>'</codeph>
+      lets you skip the column definitions of the <codeph>CREATE TABLE</codeph> statement. The column names and
+      data types are automatically configured based on the organization of the specified Parquet data file, which
+      must already reside in HDFS. You can use a data file located outside the Impala database directories, or a
+      file from an existing Impala Parquet table; either way, Impala only uses the column definitions from the file
+      and does not use the HDFS location for the <codeph>LOCATION</codeph> attribute of the new table. (Although
+      you can also specify the enclosing directory with the <codeph>LOCATION</codeph> attribute, to both use the
+      same schema as the data file and point the Impala table at the associated directory for querying.)
+    </p>
+
+    <p rev="1.4.0">
+      The following considerations apply when you use the <codeph>CREATE TABLE LIKE PARQUET</codeph> technique:
+    </p>
+
+    <ul rev="1.4.0">
+      <li>
+        Any column comments from the original table are not preserved in the new table. Each column in the new
+        table has a comment stating the low-level Parquet field type used to deduce the appropriate SQL column
+        type.
+      </li>
+
+      <li>
+        If you use a data file from a partitioned Impala table, any partition key columns from the original table
+        are left out of the new table, because they are represented in HDFS directory names rather than stored in
+        the data file. To preserve the partition information, repeat the same <codeph>PARTITION</codeph> clause as
+        in the original <codeph>CREATE TABLE</codeph> statement.
+      </li>
+
+      <li>
+        The file format of the new table defaults to text, as with other kinds of <codeph>CREATE TABLE</codeph>
+        statements. To make the new table also use Parquet format, include the clause <codeph>STORED AS
+        PARQUET</codeph> in the <codeph>CREATE TABLE LIKE PARQUET</codeph> statement.
+      </li>
+
+      <li>
+        If the Parquet data file comes from an existing Impala table, currently, any <codeph>TINYINT</codeph> or
+        <codeph>SMALLINT</codeph> columns are turned into <codeph>INT</codeph> columns in the new table.
+        Internally, Parquet stores such values as 32-bit integers.
+      </li>
+
+      <li>
+        When the destination table uses the Parquet file format, the <codeph>CREATE TABLE AS SELECT</codeph> and
+        <codeph>INSERT ... SELECT</codeph> statements always create at least one data file, even if the
+        <codeph>SELECT</codeph> part of the statement does not match any rows. You can use such an empty Parquet
+        data file as a template for subsequent <codeph>CREATE TABLE LIKE PARQUET</codeph> statements.
+      </li>
+    </ul>
+
+    <p>
+      For more details about creating Parquet tables, and examples of the <codeph>CREATE TABLE LIKE
+      PARQUET</codeph> syntax, see <xref href="impala_parquet.xml#parquet"/>.
+    </p>
+
+    <p>
+      <b>Visibility and Metadata (TBLPROPERTIES and WITH SERDEPROPERTIES clauses):</b>
+    </p>
+
+    <p rev="1.2">
+      You can associate arbitrary items of metadata with a table by specifying the <codeph>TBLPROPERTIES</codeph>
+      clause. This clause takes a comma-separated list of key-value pairs and stores those items in the metastore
+      database. You can also change the table properties later with an <codeph>ALTER TABLE</codeph> statement. You
+      can observe the table properties for different delimiter and escape characters using the <codeph>DESCRIBE
+      FORMATTED</codeph> command, and change those settings for an existing table with <codeph>ALTER TABLE ... SET
+      TBLPROPERTIES</codeph>.
+    </p>
+
+    <p rev="1.2">
+      You can also associate SerDes properties with the table by specifying key-value pairs through the
+      <codeph>WITH SERDEPROPERTIES</codeph> clause. This metadata is not used by Impala, which has its own built-in
+      serializer and deserializer for the file formats it supports. Particular property values might be needed for
+      Hive compatibility with certain variations of file formats, particularly Avro.
+    </p>
+
+    <p>
+      Some DDL operations that interact with other Hadoop components require specifying particular values in the
+      <codeph>SERDEPROPERTIES</codeph> or <codeph>TBLPROPERTIES</codeph> fields, such as creating an Avro table or
+      an HBase table. (You typically create HBase tables in Hive, because they require additional clauses not
+      currently available in Impala.)
+<!-- Haven't got a working example from Lenni, so suppressing this recommendation for now.
+        The Avro schema properties can be specified through either
+        <codeph>TBLPROPERTIES</codeph> or <codeph>SERDEPROPERTIES</codeph>;
+        for best compatibility with future versions of Hive,
+        use <codeph>SERDEPROPERTIES</codeph> in this case.
+-->
+    </p>
+
+    <p>
+      To see the column definitions and column comments for an existing table, for example before issuing a
+      <codeph>CREATE TABLE ... LIKE</codeph> or a <codeph>CREATE TABLE ... AS SELECT</codeph> statement, issue the
+      statement <codeph>DESCRIBE <varname>table_name</varname></codeph>. To see even more detail, such as the
+      location of data files and the values for clauses such as <codeph>ROW FORMAT</codeph> and <codeph>STORED
+      AS</codeph>, issue the statement <codeph>DESCRIBE FORMATTED <varname>table_name</varname></codeph>.
+      <codeph>DESCRIBE FORMATTED</codeph> is also needed to see any overall table comment (as opposed to individual
+      column comments).
+    </p>
+
+    <p>
+      After creating a table, your <cmdname>impala-shell</cmdname> session or another
+      <cmdname>impala-shell</cmdname> connected to the same node can immediately query that table. There might be a
+      brief interval (one statestore heartbeat) before the table can be queried through a different Impala node. To
+      make the <codeph>CREATE TABLE</codeph> statement return only when the table is recognized by all Impala nodes
+      in the cluster, enable the <codeph>SYNC_DDL</codeph> query option.
+    </p>
+
+    <p rev="1.4.0">
+      <b>HDFS caching (CACHED IN clause):</b>
+    </p>
+
+    <p rev="1.4.0">
+      If you specify the <codeph>CACHED IN</codeph> clause, any existing or future data files in the table
+      directory or the partition subdirectories are designated to be loaded into memory with the HDFS caching
+      mechanism. See <xref href="impala_perf_hdfs_caching.xml#hdfs_caching"/> for details about using the HDFS
+      caching feature.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/impala_cache_replication_factor"/>
+
+<!-- Say something in here about the SHOW statement, e.g. SHOW TABLES, SHOW TABLE/COLUMN STATS, SHOW PARTITIONS. -->
+
+    <p>
+      <b>Column order</b>:
+    </p>
+
+    <p>
+      If you intend to use the table to hold data files produced by some external source, specify the columns in
+      the same order as they appear in the data files.
+    </p>
+
+    <p>
+      If you intend to insert or copy data into the table through Impala, or if you have control over the way
+      externally produced data files are arranged, use your judgment to specify columns in the most convenient
+      order:
+    </p>
+
+    <ul>
+      <li>
+        <p>
+          If certain columns are often <codeph>NULL</codeph>, specify those columns last. You might produce data
+          files that omit these trailing columns entirely. Impala automatically fills in the <codeph>NULL</codeph>
+          values if so.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          If an unpartitioned table will be used as the source for an <codeph>INSERT ... SELECT</codeph> operation
+          into a partitioned table, specify last in the unpartitioned table any columns that correspond to
+          partition key columns in the partitioned table, and in the same order as the partition key columns are
+          declared in the partitioned table. This technique lets you use <codeph>INSERT ... SELECT *</codeph> when
+          copying data to the partitioned table, rather than specifying each column name individually.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          If you specify columns in an order that you later discover is suboptimal, you can sometimes work around
+          the problem without recreating the table. You can create a view that selects columns from the original
+          table in a permuted order, then do a <codeph>SELECT *</codeph> from the view. When inserting data into a
+          table, you can specify a permuted order for the inserted columns to match the order in the destination
+          table.
+        </p>
+      </li>
+    </ul>
+
+    <p conref="../shared/impala_common.xml#common/hive_blurb"/>
+
+    <p>
+      Impala queries can make use of metadata about the table and columns, such as the number of rows in a table or
+      the number of different values in a column. Prior to Impala 1.2.2, to create this metadata, you issued the
+      <codeph>ANALYZE TABLE</codeph> statement in Hive to gather this information, after creating the table and
+      loading representative data into it. In Impala 1.2.2 and higher, the <codeph>COMPUTE STATS</codeph> statement
+      produces these statistics within Impala, without needing to use Hive at all.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/hbase_blurb"/>
+
+    <note>
+      <p>
+        The Impala <codeph>CREATE TABLE</codeph> statement cannot create an HBase table, because it currently does
+        not support the <codeph>STORED BY</codeph> clause needed for HBase tables. Create such tables in Hive, then
+        query them through Impala. For information on using Impala with HBase tables, see
+        <xref href="impala_hbase.xml#impala_hbase"/>.
+      </p>
+    </note>
+
+    <p conref="../shared/impala_common.xml#common/s3_blurb"/>
+    <p rev="2.2.0">
+      To create a table where the data resides in the Amazon Simple Storage Service (S3),
+      specify a <codeph>s3a://</codeph> prefix <codeph>LOCATION</codeph> attribute pointing to the data files in S3.
+    </p>
+
+    <p rev="2.6.0 CDH-39913 IMPALA-1878">
+      In <keyword keyref="impala26_full"/> and higher, you can
+      use this special <codeph>LOCATION</codeph> syntax
+      as part of a <codeph>CREATE TABLE AS SELECT</codeph> statement.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/s3_ddl"/>
+
+    <p conref="../shared/impala_common.xml#common/insert_sort_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/hdfs_blurb"/>
+
+    <p>
+      The <codeph>CREATE TABLE</codeph> statement for an internal table creates a directory in HDFS. The
+      <codeph>CREATE EXTERNAL TABLE</codeph> statement associates the table with an existing HDFS directory, and
+      does not create any new directory in HDFS. To locate the HDFS data directory for a table, issue a
+      <codeph>DESCRIBE FORMATTED <varname>table</varname></codeph> statement. To examine the contents of that HDFS
+      directory, use an OS command such as <codeph>hdfs dfs -ls hdfs://<varname>path</varname></codeph>, either
+      from the OS command line or through the <codeph>shell</codeph> or <codeph>!</codeph> commands in
+      <cmdname>impala-shell</cmdname>.
+    </p>
+
+    <p>
+      The <codeph>CREATE TABLE AS SELECT</codeph> syntax creates data files under the table data directory to hold
+      any data copied by the <codeph>INSERT</codeph> portion of the statement. (Even if no data is copied, Impala
+      might create one or more empty data files.)
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+    <p rev="CDH-19187">
+      The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+      typically the <codeph>impala</codeph> user, must have both execute and write
+      permission for the database directory where the table is being created.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/security_blurb"/>
+    <p conref="../shared/impala_common.xml#common/redaction_yes"/>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_maybe"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_tables.xml#tables"/>,
+      <xref href="impala_alter_table.xml#alter_table"/>, <xref href="impala_drop_table.xml#drop_table"/>,
+      <xref href="impala_partitioning.xml#partitioning"/>, <xref href="impala_tables.xml#internal_tables"/>,
+      <xref href="impala_tables.xml#external_tables"/>, <xref href="impala_compute_stats.xml#compute_stats"/>,
+      <xref href="impala_sync_ddl.xml#sync_ddl"/>, <xref href="impala_show.xml#show_tables"/>,
+      <xref href="impala_show.xml#show_create_table"/>, <xref href="impala_describe.xml#describe"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_create_view.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_create_view.xml b/docs/topics/impala_create_view.xml
new file mode 100644
index 0000000..446acf2
--- /dev/null
+++ b/docs/topics/impala_create_view.xml
@@ -0,0 +1,139 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.1" id="create_view">
+
+  <title>CREATE VIEW Statement</title>
+  <titlealts audience="PDF"><navtitle>CREATE VIEW</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Schemas"/>
+      <data name="Category" value="Views"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">CREATE VIEW statement</indexterm>
+      The <codeph>CREATE VIEW</codeph> statement lets you create a shorthand abbreviation for a more complicated
+      query. The base query can involve joins, expressions, reordered columns, column aliases, and other SQL
+      features that can make a query hard to understand or maintain.
+    </p>
+
+    <p>
+      Because a view is purely a logical construct (an alias for a query) with no physical data behind it,
+      <codeph>ALTER VIEW</codeph> only involves changes to metadata in the metastore database, not any data files
+      in HDFS.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>CREATE VIEW [IF NOT EXISTS] <varname>view_name</varname> [(<varname>column_list</varname>)]
+  AS <varname>select_statement</varname></codeblock>
+
+    <p conref="../shared/impala_common.xml#common/ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      The <codeph>CREATE VIEW</codeph> statement can be useful in scenarios such as the following:
+    </p>
+
+    <ul>
+      <li>
+        To turn even the most lengthy and complicated SQL query into a one-liner. You can issue simple queries
+        against the view from applications, scripts, or interactive queries in <cmdname>impala-shell</cmdname>.
+        For example:
+<codeblock>select * from <varname>view_name</varname>;
+select * from <varname>view_name</varname> order by c1 desc limit 10;</codeblock>
+        The more complicated and hard-to-read the original query, the more benefit there is to simplifying the
+        query using a view.
+      </li>
+
+      <li>
+        To hide the underlying table and column names, to minimize maintenance problems if those names change. In
+        that case, you re-create the view using the new names, and all queries that use the view rather than the
+        underlying tables keep running with no changes.
+      </li>
+
+      <li>
+        To experiment with optimization techniques and make the optimized queries available to all applications.
+        For example, if you find a combination of <codeph>WHERE</codeph> conditions, join order, join hints, and so
+        on that works the best for a class of queries, you can establish a view that incorporates the
+        best-performing techniques. Applications can then make relatively simple queries against the view, without
+        repeating the complicated and optimized logic over and over. If you later find a better way to optimize the
+        original query, when you re-create the view, all the applications immediately take advantage of the
+        optimized base query.
+      </li>
+
+      <li>
+        To simplify a whole class of related queries, especially complicated queries involving joins between
+        multiple tables, complicated expressions in the column list, and other SQL syntax that makes the query
+        difficult to understand and debug. For example, you might create a view that joins several tables, filters
+        using several <codeph>WHERE</codeph> conditions, and selects several columns from the result set.
+        Applications might issue queries against this view that only vary in their <codeph>LIMIT</codeph>,
+        <codeph>ORDER BY</codeph>, and similar simple clauses.
+      </li>
+    </ul>
+
+    <p>
+      For queries that require repeating complicated clauses over and over again, for example in the select list,
+      <codeph>ORDER BY</codeph>, and <codeph>GROUP BY</codeph> clauses, you can use the <codeph>WITH</codeph>
+      clause as an alternative to creating a view.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+    <p conref="../shared/impala_common.xml#common/complex_types_views"/>
+    <p conref="../shared/impala_common.xml#common/complex_types_views_caveat"/>
+
+    <p conref="../shared/impala_common.xml#common/sync_ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/security_blurb"/>
+    <p conref="../shared/impala_common.xml#common/redaction_yes"/>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<!-- TK: Elaborate on these, show queries and real output. -->
+
+<codeblock>-- Create a view that is exactly the same as the underlying table.
+create view v1 as select * from t1;
+
+-- Create a view that includes only certain columns from the underlying table.
+create view v2 as select c1, c3, c7 from t1;
+
+-- Create a view that filters the values from the underlying table.
+create view v3 as select distinct c1, c3, c7 from t1 where c1 is not null and c5 &gt; 0;
+
+-- Create a view that that reorders and renames columns from the underlying table.
+create view v4 as select c4 as last_name, c6 as address, c2 as birth_date from t1;
+
+-- Create a view that runs functions to convert or transform certain columns.
+create view v5 as select c1, cast(c3 as string) c3, concat(c4,c5) c5, trim(c6) c6, "Constant" c8 from t1;
+
+-- Create a view that hides the complexity of a view query.
+create view v6 as select t1.c1, t2.c2 from t1 join t2 on t1.id = t2.id;
+</codeblock>
+
+<!-- These examples show CREATE VIEW and corresponding DROP VIEW statements, with different combinations
+     of qualified and unqualified names. -->
+
+    <p conref="../shared/impala_common.xml#common/create_drop_view_examples"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_views.xml#views"/>, <xref href="impala_alter_view.xml#alter_view"/>,
+      <xref href="impala_drop_view.xml#drop_view"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_data_sources.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_data_sources.xml b/docs/topics/impala_data_sources.xml
new file mode 100644
index 0000000..8fb46e1
--- /dev/null
+++ b/docs/topics/impala_data_sources.xml
@@ -0,0 +1,22 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.4.0" id="data_sources">
+
+  <title>Data Sources</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <xref href="impala_create_data_source.xml#create_data_source"/>
+      <xref href="impala_drop_data_source.xml#drop_data_source"/>
+      <xref href="impala_create_table.xml#create_table"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_databases.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_databases.xml b/docs/topics/impala_databases.xml
new file mode 100644
index 0000000..3253e75
--- /dev/null
+++ b/docs/topics/impala_databases.xml
@@ -0,0 +1,65 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="databases">
+
+  <title>Overview of Impala Databases</title>
+  <titlealts audience="PDF"><navtitle>Databases</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Databases"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      In Impala, a database is a logical container for a group of tables. Each database defines a separate
+      namespace. Within a database, you can refer to the tables inside it using their unqualified names. Different
+      databases can contain tables with identical names.
+    </p>
+
+    <p>
+      Creating a database is a lightweight operation. There are minimal database-specific properties to configure,
+      only <codeph>LOCATION</codeph> and <codeph>COMMENT</codeph>.  There is no <codeph>ALTER DATABASE</codeph> statement.
+    </p>
+
+    <p>
+      Typically, you create a separate database for each project or application, to avoid naming conflicts between
+      tables and to make clear which tables are related to each other. The <codeph>USE</codeph> statement lets
+      you switch between databases. Unqualified references to tables, views, and functions refer to objects
+      within the current database. You can also refer to objects in other databases by using qualified names
+      of the form <codeph><varname>dbname</varname>.<varname>object_name</varname></codeph>.
+    </p>
+
+    <p>
+      Each database is physically represented by a directory in HDFS. When you do not specify a <codeph>LOCATION</codeph>
+      attribute, the directory is located in the Impala data directory with the associated tables managed by Impala.
+      When you do specify a <codeph>LOCATION</codeph> attribute, any read and write operations for tables in that
+      database are relative to the specified HDFS directory.
+    </p>
+
+    <p>
+      There is a special database, named <codeph>default</codeph>, where you begin when you connect to Impala.
+      Tables created in <codeph>default</codeph> are physically located one level higher in HDFS than all the
+      user-created databases.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/builtins_db"/>
+
+    <p>
+      <b>Related statements:</b>
+    </p>
+
+    <p>
+      <xref href="impala_create_database.xml#create_database"/>,
+      <xref href="impala_drop_database.xml#drop_database"/>, <xref href="impala_use.xml#use"/>,
+      <xref href="impala_show.xml#show_databases"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_datatypes.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_datatypes.xml b/docs/topics/impala_datatypes.xml
new file mode 100644
index 0000000..e45867e
--- /dev/null
+++ b/docs/topics/impala_datatypes.xml
@@ -0,0 +1,43 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="datatypes">
+
+  <title>Data Types</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">data types</indexterm>
+      Impala supports a set of data types that you can use for table columns, expression values, and function
+      arguments and return values.
+    </p>
+
+    <note>
+      Currently, Impala supports only scalar types, not composite or nested types. Accessing a table containing any
+      columns with unsupported types causes an error.
+    </note>
+
+    <p outputclass="toc"/>
+
+    <p>
+      For the notation to write literals of each of these data types, see
+      <xref href="impala_literals.xml#literals"/>.
+    </p>
+
+    <p>
+      See <xref href="impala_langref_unsupported.xml#langref_hiveql_delta"/> for differences between Impala and
+      Hive data types.
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_date.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_date.xml b/docs/topics/impala_date.xml
new file mode 100644
index 0000000..ae17b9a
--- /dev/null
+++ b/docs/topics/impala_date.xml
@@ -0,0 +1,104 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept audience="Cloudera" id="date" rev="2.0.0">
+
+  <title>DATE Data Type (<keyword keyref="impala21"/> or higher only)</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Dates and Times"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">DATE data type</indexterm>
+      A type representing the date (year, month, and day) as a single numeric value. Used to represent a broader
+      date range than possible with the <codeph>TIMESTAMP</codeph> type, with fewer distinct values than
+      <codeph>TIMESTAMP</codeph>, and in a more compact and efficient form than using a <codeph>STRING</codeph>
+      such as <codeph>'2014-12-31'</codeph>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock><varname>column_name</varname> DATE</codeblock>
+
+    <p>
+      <b>Range:</b> January 1, -4712 BC .. December 31, 9999 AD.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/hbase_ok"/>
+
+    <p conref="../shared/impala_common.xml#common/parquet_blurb"/>
+
+    <ul>
+      <li>
+        This type can be read from and written to Parquet files.
+      </li>
+
+      <li>
+        There is no requirement for a particular level of Parquet.
+      </li>
+
+      <li>
+        Parquet files generated by Impala and containing this type can be freely interchanged with other components
+        such as Hive and MapReduce.
+      </li>
+    </ul>
+
+    <p conref="../shared/impala_common.xml#common/hive_blurb"/>
+
+    <p>
+      TK.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/conversion_blurb"/>
+
+    <p>
+      TK.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/partitioning_blurb"/>
+
+    <p>
+      This type can be used for partition key columns. Because it has less granularity (and thus fewer distinct
+      values) than an equivalent <codeph>TIMESTAMP</codeph> column, and numeric columns are more efficient as
+      partition keys than strings, prefer to partition by a <codeph>DATE</codeph> column rather than a
+      <codeph>TIMESTAMP</codeph> column or a <codeph>STRING</codeph> representation of a date.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/compatibility_blurb"/>
+
+    <p>
+      This type is available on CDH 5.2 or higher.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/internals_2_bytes"/>
+
+    <p conref="../shared/impala_common.xml#common/added_in_20"/>
+
+    <p conref="../shared/impala_common.xml#common/column_stats_constant"/>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+    <p>
+      Things happen when converting <codeph>TIMESTAMP</codeph> to <codeph>DATE</codeph> or <codeph>DATE</codeph> to
+      <codeph>TIMESTAMP</codeph>. TK.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      The <xref href="impala_timestamp.xml#timestamp">TIMESTAMP</xref> data type is closely related. Some functions
+      from <xref href="impala_datetime_functions.xml#datetime_functions"/> accept and return <codeph>DATE</codeph>
+      values.
+    </p>
+  </conbody>
+</concept>


[26/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_hbase.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_hbase.xml b/docs/topics/impala_hbase.xml
new file mode 100644
index 0000000..8b8d013
--- /dev/null
+++ b/docs/topics/impala_hbase.xml
@@ -0,0 +1,904 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="impala_hbase">
+
+  <title id="hbase">Using Impala to Query HBase Tables</title>
+  <titlealts audience="PDF"><navtitle>HBase Tables</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="HBase"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Tables"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">HBase</indexterm>
+      You can use Impala to query HBase tables. This capability allows convenient access to a storage system that
+      is tuned for different kinds of workloads than the default with Impala. The default Impala tables use data
+      files stored on HDFS, which are ideal for bulk loads and queries using full-table scans. In contrast, HBase
+      can do efficient queries for data organized for OLTP-style workloads, with lookups of individual rows or
+      ranges of values.
+    </p>
+
+    <p>
+      From the perspective of an Impala user, coming from an RDBMS background, HBase is a kind of key-value store
+      where the value consists of multiple fields. The key is mapped to one column in the Impala table, and the
+      various fields of the value are mapped to the other columns in the Impala table.
+    </p>
+
+    <p>
+      For background information on HBase, see the snapshot of the Apache HBase site (including documentation) for
+      the level of HBase that comes with
+      <xref href="https://archive.cloudera.com/cdh5/cdh/5/hbase/" scope="external" format="html">CDH 5</xref>. To
+      install HBase on a CDH cluster, see the installation instructions for
+      <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_hbase_installation.html" scope="external" format="html">CDH 5</xref>.
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="hbase_using">
+
+    <title>Overview of Using HBase with Impala</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Concepts"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        When you use Impala with HBase:
+      </p>
+
+      <ul>
+        <li>
+          You create the tables on the Impala side using the Hive shell, because the Impala <codeph>CREATE
+          TABLE</codeph> statement currently does not support custom SerDes and some other syntax needed for these
+          tables:
+          <ul>
+            <li>
+              You designate it as an HBase table using the <codeph>STORED BY
+              'org.apache.hadoop.hive.hbase.HBaseStorageHandler'</codeph> clause on the Hive <codeph>CREATE
+              TABLE</codeph> statement.
+            </li>
+
+            <li>
+              You map these specially created tables to corresponding tables that exist in HBase, with the clause
+              <codeph>TBLPROPERTIES("hbase.table.name" = "<varname>table_name_in_hbase</varname>")</codeph> on the
+              Hive <codeph>CREATE TABLE</codeph> statement.
+            </li>
+
+            <li>
+              See <xref href="#hbase_queries"/> for a full example.
+            </li>
+          </ul>
+        </li>
+
+        <li>
+          You define the column corresponding to the HBase row key as a string with the <codeph>#string</codeph>
+          keyword, or map it to a <codeph>STRING</codeph> column.
+        </li>
+
+        <li>
+          Because Impala and Hive share the same metastore database, once you create the table in Hive, you can
+          query or insert into it through Impala. (After creating a new table through Hive, issue the
+          <codeph>INVALIDATE METADATA</codeph> statement in <cmdname>impala-shell</cmdname> to make Impala aware of
+          the new table.)
+        </li>
+
+        <li>
+          You issue queries against the Impala tables. For efficient queries, use <codeph>WHERE</codeph> clauses to
+          find a single key value or a range of key values wherever practical, by testing the Impala column
+          corresponding to the HBase row key. Avoid queries that do full-table scans, which are efficient for
+          regular Impala tables but inefficient in HBase.
+        </li>
+      </ul>
+
+      <p>
+        To work with an HBase table from Impala, ensure that the <codeph>impala</codeph> user has read/write
+        privileges for the HBase table, using the <codeph>GRANT</codeph> command in the HBase shell. For details
+        about HBase security, see the
+        <xref href="http://hbase.apache.org/book/ch08s04.html" format="html" scope="external">Security chapter in
+        the HBase Reference Guide</xref>.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="hbase_config">
+
+    <title>Configuring HBase for Use with Impala</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Configuring"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        HBase works out of the box with Impala. There is no mandatory configuration needed to use these two
+        components together.
+      </p>
+
+      <p>
+        To avoid delays if HBase is unavailable during Impala startup or after an <codeph>INVALIDATE
+        METADATA</codeph> statement, <ph rev="upstream">Cloudera</ph> recommends setting timeout values as follows in
+        <filepath>/etc/impala/conf/hbase-site.xml</filepath> (for environments not managed by Cloudera Manager):
+      </p>
+
+<codeblock>&lt;property&gt;
+  &lt;name&gt;hbase.client.retries.number&lt;/name&gt;
+  &lt;value&gt;3&lt;/value&gt;
+&lt;/property&gt;
+&lt;property&gt;
+  &lt;name&gt;hbase.rpc.timeout&lt;/name&gt;
+  &lt;value&gt;3000&lt;/value&gt;
+&lt;/property&gt;
+</codeblock>
+
+      <p>
+        Currently, Cloudera Manager does not have an Impala-only override for HBase settings, so any HBase
+        configuration change you make through Cloudera Manager would take affect for all HBase applications.
+        Therefore, this change is not recommended on systems managed by Cloudera Manager.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="hbase_types">
+
+    <title>Supported Data Types for HBase Columns</title>
+
+    <conbody>
+
+      <p>
+        To understand how Impala column data types are mapped to fields in HBase, you should have some background
+        knowledge about HBase first. You set up the mapping by running the <codeph>CREATE TABLE</codeph> statement
+        in the Hive shell. See
+        <xref href="https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration" scope="external" format="html">the
+        Hive wiki</xref> for a starting point, and <xref href="#hbase_queries"/> for examples.
+      </p>
+
+      <p>
+        HBase works as a kind of <q>bit bucket</q>, in the sense that HBase does not enforce any typing for the
+        key or value fields. All the type enforcement is done on the Impala side.
+      </p>
+
+      <p>
+        For best performance of Impala queries against HBase tables, most queries will perform comparisons in the
+        <codeph>WHERE</codeph> against the column that corresponds to the HBase row key. When creating the table
+        through the Hive shell, use the <codeph>STRING</codeph> data type for the column that corresponds to the
+        HBase row key. Impala can translate conditional tests (through operators such as <codeph>=</codeph>,
+        <codeph>&lt;</codeph>, <codeph>BETWEEN</codeph>, and <codeph>IN</codeph>) against this column into fast
+        lookups in HBase, but this optimization (<q>predicate pushdown</q>) only works when that column is
+        defined as <codeph>STRING</codeph>.
+      </p>
+
+      <p>
+        Starting in Impala 1.1, Impala also supports reading and writing to columns that are defined in the Hive
+        <codeph>CREATE TABLE</codeph> statement using binary data types, represented in the Hive table definition
+        using the <codeph>#binary</codeph> keyword, often abbreviated as <codeph>#b</codeph>. Defining numeric
+        columns as binary can reduce the overall data volume in the HBase tables. You should still define the
+        column that corresponds to the HBase row key as a <codeph>STRING</codeph>, to allow fast lookups using
+        those columns.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="hbase_performance">
+
+    <title>Performance Considerations for the Impala-HBase Integration</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Performance"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        To understand the performance characteristics of SQL queries against data stored in HBase, you should have
+        some background knowledge about how HBase interacts with SQL-oriented systems first. See
+        <xref href="https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration" scope="external" format="html">the
+        Hive wiki</xref> for a starting point; because Impala shares the same metastore database as Hive, the
+        information about mapping columns from Hive tables to HBase tables is generally applicable to Impala too.
+      </p>
+
+      <p>
+        Impala uses the HBase client API via Java Native Interface (JNI) to query data stored in HBase. This
+        querying does not read HFiles directly. The extra communication overhead makes it important to choose what
+        data to store in HBase or in HDFS, and construct efficient queries that can retrieve the HBase data
+        efficiently:
+      </p>
+
+      <ul>
+        <li>
+          Use HBase table for queries that return a single row or a range of rows, not queries that scan the entire
+          table. (If a query has no <codeph>WHERE</codeph> clause, that is a strong indicator that it is an
+          inefficient query for an HBase table.)
+        </li>
+
+        <li>
+          If you have join queries that do aggregation operations on large fact tables and join the results against
+          small dimension tables, consider using Impala for the fact tables and HBase for the dimension tables.
+          (Because Impala does a full scan on the HBase table in this case, rather than doing single-row HBase
+          lookups based on the join column, only use this technique where the HBase table is small enough that
+          doing a full table scan does not cause a performance bottleneck for the query.)
+        </li>
+      </ul>
+
+      <p>
+        Query predicates are applied to row keys as start and stop keys, thereby limiting the scope of a particular
+        lookup. If row keys are not mapped to string columns, then ordering is typically incorrect and comparison
+        operations do not work. For example, if row keys are not mapped to string columns, evaluating for greater
+        than (&gt;) or less than (&lt;) cannot be completed.
+      </p>
+
+      <p>
+        Predicates on non-key columns can be sent to HBase to scan as <codeph>SingleColumnValueFilters</codeph>,
+        providing some performance gains. In such a case, HBase returns fewer rows than if those same predicates
+        were applied using Impala. While there is some improvement, it is not as great when start and stop rows are
+        used. This is because the number of rows that HBase must examine is not limited as it is when start and
+        stop rows are used. As long as the row key predicate only applies to a single row, HBase will locate and
+        return that row. Conversely, if a non-key predicate is used, even if it only applies to a single row, HBase
+        must still scan the entire table to find the correct result.
+      </p>
+
+      <example>
+
+        <title>Interpreting EXPLAIN Output for HBase Queries</title>
+
+        <p>
+          For example, here are some queries against the following Impala table, which is mapped to an HBase table.
+          The examples show excerpts from the output of the <codeph>EXPLAIN</codeph> statement, demonstrating what
+          things to look for to indicate an efficient or inefficient query against an HBase table.
+        </p>
+
+        <p>
+          The first column (<codeph>cust_id</codeph>) was specified as the key column in the <codeph>CREATE
+          EXTERNAL TABLE</codeph> statement; for performance, it is important to declare this column as
+          <codeph>STRING</codeph>. Other columns, such as <codeph>BIRTH_YEAR</codeph> and
+          <codeph>NEVER_LOGGED_ON</codeph>, are also declared as <codeph>STRING</codeph>, rather than their
+          <q>natural</q> types of <codeph>INT</codeph> or <codeph>BOOLEAN</codeph>, because Impala can optimize
+          those types more effectively in HBase tables. For comparison, we leave one column,
+          <codeph>YEAR_REGISTERED</codeph>, as <codeph>INT</codeph> to show that filtering on this column is
+          inefficient.
+        </p>
+
+<codeblock>describe hbase_table;
+Query: describe hbase_table
++-----------------------+--------+---------+
+| name                  | type   | comment |
++-----------------------+--------+---------+
+| cust_id               | <b>string</b> |         |
+| birth_year            | <b>string</b> |         |
+| never_logged_on       | <b>string</b> |         |
+| private_email_address | string |         |
+| year_registered       | <b>int</b>    |         |
++-----------------------+--------+---------+
+</codeblock>
+
+        <p>
+          The best case for performance involves a single row lookup using an equality comparison on the column
+          defined as the row key:
+        </p>
+
+<codeblock>explain select count(*) from hbase_table where cust_id = 'some_user@example.com';
++------------------------------------------------------------------------------------+
+| Explain String                                                                     |
++------------------------------------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=1.01GB VCores=1                            |
+| WARNING: The following tables are missing relevant table and/or column statistics. |
+| hbase.hbase_table                                                                  |
+|                                                                                    |
+| 03:AGGREGATE [MERGE FINALIZE]                                                      |
+| |  output: sum(count(*))                                                           |
+| |                                                                                  |
+| 02:EXCHANGE [PARTITION=UNPARTITIONED]                                              |
+| |                                                                                  |
+| 01:AGGREGATE                                                                       |
+| |  output: count(*)                                                                |
+| |                                                                                  |
+<b>| 00:SCAN HBASE [hbase.hbase_table]                                                  |</b>
+<b>|    start key: some_user@example.com                                                |</b>
+<b>|    stop key: some_user@example.com\0                                               |</b>
++------------------------------------------------------------------------------------+
+</codeblock>
+
+        <p>
+          Another type of efficient query involves a range lookup on the row key column, using SQL operators such
+          as greater than (or equal), less than (or equal), or <codeph>BETWEEN</codeph>. This example also includes
+          an equality test on a non-key column; because that column is a <codeph>STRING</codeph>, Impala can let
+          HBase perform that test, indicated by the <codeph>hbase filters:</codeph> line in the
+          <codeph>EXPLAIN</codeph> output. Doing the filtering within HBase is more efficient than transmitting all
+          the data to Impala and doing the filtering on the Impala side.
+        </p>
+
+<codeblock>explain select count(*) from hbase_table where cust_id between 'a' and 'b'
+  and never_logged_on = 'true';
++------------------------------------------------------------------------------------+
+| Explain String                                                                     |
++------------------------------------------------------------------------------------+
+...
+<!--| Estimated Per-Host Requirements: Memory=1.01GB VCores=1                            |
+| WARNING: The following tables are missing relevant table and/or column statistics. |
+| hbase.hbase_table                                                                  |
+|                                                                                    |
+| 03:AGGREGATE [MERGE FINALIZE]                                                      |
+| |  output: sum(count(*))                                                           |
+| |                                                                                  |
+| 02:EXCHANGE [PARTITION=UNPARTITIONED]                                              |
+| |                                                                                  |-->
+| 01:AGGREGATE                                                                       |
+| |  output: count(*)                                                                |
+| |                                                                                  |
+<b>| 00:SCAN HBASE [hbase.hbase_table]                                                  |</b>
+<b>|    start key: a                                                                    |</b>
+<b>|    stop key: b\0                                                                   |</b>
+<b>|    hbase filters: cols:never_logged_on EQUAL 'true'                                |</b>
++------------------------------------------------------------------------------------+
+</codeblock>
+
+        <p>
+          The query is less efficient if Impala has to evaluate any of the predicates, because Impala must scan the
+          entire HBase table. Impala can only push down predicates to HBase for columns declared as
+          <codeph>STRING</codeph>. This example tests a column declared as <codeph>INT</codeph>, and the
+          <codeph>predicates:</codeph> line in the <codeph>EXPLAIN</codeph> output indicates that the test is
+          performed after the data is transmitted to Impala.
+        </p>
+
+<codeblock>explain select count(*) from hbase_table where year_registered = 2010;
++------------------------------------------------------------------------------------+
+| Explain String                                                                     |
++------------------------------------------------------------------------------------+
+...
+<!--| Estimated Per-Host Requirements: Memory=1.01GB VCores=1                            |
+| WARNING: The following tables are missing relevant table and/or column statistics. |
+| hbase.hbase_table                                                                  |
+|                                                                                    |
+| 03:AGGREGATE [MERGE FINALIZE]                                                      |
+| |  output: sum(count(*))                                                           |
+| |                                                                                  |
+| 02:EXCHANGE [PARTITION=UNPARTITIONED]                                              |
+| |                                                                                  |-->
+| 01:AGGREGATE                                                                       |
+| |  output: count(*)                                                                |
+| |                                                                                  |
+<b>| 00:SCAN HBASE [hbase.hbase_table]                                                  |</b>
+<b>|    predicates: year_registered = 2010                                              |</b>
++------------------------------------------------------------------------------------+
+</codeblock>
+
+        <p>
+          The same inefficiency applies if the key column is compared to any non-constant value. Here, even though
+          the key column is a <codeph>STRING</codeph>, and is tested using an equality operator, Impala must scan
+          the entire HBase table because the key column is compared to another column value rather than a constant.
+        </p>
+
+<codeblock>explain select count(*) from hbase_table where cust_id = private_email_address;
++------------------------------------------------------------------------------------+
+| Explain String                                                                     |
++------------------------------------------------------------------------------------+
+...
+<!--| Estimated Per-Host Requirements: Memory=1.01GB VCores=1                            |
+| WARNING: The following tables are missing relevant table and/or column statistics. |
+| hbase.hbase_table                                                                  |
+|                                                                                    |
+| 03:AGGREGATE [MERGE FINALIZE]                                                      |
+| |  output: sum(count(*))                                                           |
+| |                                                                                  |
+| 02:EXCHANGE [PARTITION=UNPARTITIONED]                                              |
+| |                                                                                  |-->
+| 01:AGGREGATE                                                                       |
+| |  output: count(*)                                                                |
+| |                                                                                  |
+<b>| 00:SCAN HBASE [hbase.hbase_table]                                                  |</b>
+<b>|    predicates: cust_id = private_email_address                                    |</b>
++------------------------------------------------------------------------------------+
+</codeblock>
+
+        <p>
+          Currently, tests on the row key using <codeph>OR</codeph> or <codeph>IN</codeph> clauses are not
+          optimized into direct lookups either. Such limitations might be lifted in the future, so always check the
+          <codeph>EXPLAIN</codeph> output to be sure whether a particular SQL construct results in an efficient
+          query or not for HBase tables.
+        </p>
+
+<codeblock>explain select count(*) from hbase_table where
+  cust_id = 'some_user@example.com' or cust_id = 'other_user@example.com';
++----------------------------------------------------------------------------------------+
+| Explain String                                                                         |
++----------------------------------------------------------------------------------------+
+...
+<!--| Estimated Per-Host Requirements: Memory=1.01GB VCores=1                                |
+| WARNING: The following tables are missing relevant table and/or column statistics.     |
+| hbase.hbase_table                                                                      |
+|                                                                                        |
+| 03:AGGREGATE [MERGE FINALIZE]                                                          |
+| |  output: sum(count(*))                                                               |
+| |                                                                                      |
+| 02:EXCHANGE [PARTITION=UNPARTITIONED]                                                  |
+| |                                                                                      |-->
+| 01:AGGREGATE                                                                           |
+| |  output: count(*)                                                                    |
+| |                                                                                      |
+<b>| 00:SCAN HBASE [hbase.hbase_table]                                                      |</b>
+<b>|    predicates: cust_id = 'some_user@example.com' OR cust_id = 'other_user@example.com' |</b>
++----------------------------------------------------------------------------------------+
+
+explain select count(*) from hbase_table where
+  cust_id in ('some_user@example.com', 'other_user@example.com');
++------------------------------------------------------------------------------------+
+| Explain String                                                                     |
++------------------------------------------------------------------------------------+
+...
+<!--| Estimated Per-Host Requirements: Memory=1.01GB VCores=1                            |
+| WARNING: The following tables are missing relevant table and/or column statistics. |
+| hbase.hbase_table                                                                  |
+|                                                                                    |
+| 03:AGGREGATE [MERGE FINALIZE]                                                      |
+| |  output: sum(count(*))                                                           |
+| |                                                                                  |
+| 02:EXCHANGE [PARTITION=UNPARTITIONED]                                              |
+| |                                                                                  |-->
+| 01:AGGREGATE                                                                       |
+| |  output: count(*)                                                                |
+| |                                                                                  |
+<b>| 00:SCAN HBASE [hbase.hbase_table]                                                  |</b>
+<b>|    predicates: cust_id IN ('some_user@example.com', 'other_user@example.com')      |</b>
++------------------------------------------------------------------------------------+
+</codeblock>
+
+        <p>
+          Either rewrite into separate queries for each value and combine the results in the application, or
+          combine the single-row queries using UNION ALL:
+        </p>
+
+<codeblock>select count(*) from hbase_table where cust_id = 'some_user@example.com';
+select count(*) from hbase_table where cust_id = 'other_user@example.com';
+
+explain
+  select count(*) from hbase_table where cust_id = 'some_user@example.com'
+  union all
+  select count(*) from hbase_table where cust_id = 'other_user@example.com';
++------------------------------------------------------------------------------------+
+| Explain String                                                                     |
++------------------------------------------------------------------------------------+
+...
+<!--| Estimated Per-Host Requirements: Memory=1.01GB VCores=1                            |
+| WARNING: The following tables are missing relevant table and/or column statistics. |
+| hbase.hbase_table                                                                  |
+|                                                                                    |
+| 09:EXCHANGE [PARTITION=UNPARTITIONED]                                              |
+| |                                                                                  |
+| |&minus;&minus;11:MERGE                                                                        |
+| |  |                                                                               |
+| |  08:AGGREGATE [MERGE FINALIZE]                                                   |
+| |  |  output: sum(count(*))                                                        |
+| |  |                                                                               |
+| |  07:EXCHANGE [PARTITION=UNPARTITIONED]                                           |
+| |  |                                                                               |-->
+| |  04:AGGREGATE                                                                    |
+| |  |  output: count(*)                                                             |
+| |  |                                                                               |
+<b>| |  03:SCAN HBASE [hbase.hbase_table]                                               |</b>
+<b>| |     start key: other_user@example.com                                            |</b>
+<b>| |     stop key: other_user@example.com\0                                           |</b>
+| |                                                                                  |
+| 10:MERGE                                                                           |
+...
+<!--| |                                                                                  |
+| 06:AGGREGATE [MERGE FINALIZE]                                                      |
+| |  output: sum(count(*))                                                           |
+| |                                                                                  |
+| 05:EXCHANGE [PARTITION=UNPARTITIONED]                                              |
+| |                                                                                  |-->
+| 02:AGGREGATE                                                                       |
+| |  output: count(*)                                                                |
+| |                                                                                  |
+<b>| 01:SCAN HBASE [hbase.hbase_table]                                                  |</b>
+<b>|    start key: some_user@example.com                                                |</b>
+<b>|    stop key: some_user@example.com\0                                               |</b>
++------------------------------------------------------------------------------------+
+</codeblock>
+
+      </example>
+
+      <example>
+
+        <title>Configuration Options for Java HBase Applications</title>
+
+        <p> If you have an HBase Java application that calls the
+            <codeph>setCacheBlocks</codeph> or <codeph>setCaching</codeph>
+          methods of the class <xref
+            href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html"
+            scope="external" format="html"
+            >org.apache.hadoop.hbase.client.Scan</xref>, you can set these same
+          caching behaviors through Impala query options, to control the memory
+          pressure on the HBase RegionServer. For example, when doing queries in
+          HBase that result in full-table scans (which by default are
+          inefficient for HBase), you can reduce memory usage and speed up the
+          queries by turning off the <codeph>HBASE_CACHE_BLOCKS</codeph> setting
+          and specifying a large number for the <codeph>HBASE_CACHING</codeph>
+          setting.
+          <!-- That recommendation w.r.t. full-table scans comes from the Cloudera HBase forum: http://community.cloudera.com/t5/Realtime-Random-Access-Apache/How-to-optimise-Full-Table-Scan-FTS-in-HBase/td-p/97 -->
+        </p>
+
+        <p>
+          To set these options, issue commands like the following in <cmdname>impala-shell</cmdname>:
+        </p>
+
+<codeblock>-- Same as calling setCacheBlocks(true) or setCacheBlocks(false).
+set hbase_cache_blocks=true;
+set hbase_cache_blocks=false;
+
+-- Same as calling setCaching(rows).
+set hbase_caching=1000;
+</codeblock>
+
+        <p>
+          Or update the <cmdname>impalad</cmdname> defaults file <filepath>/etc/default/impala</filepath> and
+          include settings for <codeph>HBASE_CACHE_BLOCKS</codeph> and/or <codeph>HBASE_CACHING</codeph> in the
+          <codeph>-default_query_options</codeph> setting for <codeph>IMPALA_SERVER_ARGS</codeph>. See
+          <xref href="impala_config_options.xml#config_options"/> for details.
+        </p>
+
+        <note>
+          In Impala 2.0 and later, these options are settable through the JDBC or ODBC interfaces using the
+          <codeph>SET</codeph> statement.
+        </note>
+
+      </example>
+    </conbody>
+  </concept>
+
+  <concept id="hbase_scenarios">
+
+    <title>Use Cases for Querying HBase through Impala</title>
+    <prolog>
+      <metadata>
+        <data name="Category" value="Use Cases"/>
+      </metadata>
+    </prolog>
+
+    <conbody>
+
+      <p>
+        The following are popular use cases for using Impala to query HBase tables:
+      </p>
+
+      <ul>
+        <li>
+          Keeping large fact tables in Impala, and smaller dimension tables in HBase. The fact tables use Parquet
+          or other binary file format optimized for scan operations. Join queries scan through the large Impala
+          fact tables, and cross-reference the dimension tables using efficient single-row lookups in HBase.
+        </li>
+
+        <li>
+          Using HBase to store rapidly incrementing counters, such as how many times a web page has been viewed, or
+          on a social network, how many connections a user has or how many votes a post received. HBase is
+          efficient for capturing such changeable data: the append-only storage mechanism is efficient for writing
+          each change to disk, and a query always returns the latest value. An application could query specific
+          totals like these from HBase, and combine the results with a broader set of data queried from Impala.
+        </li>
+
+        <li>
+          <p>
+            Storing very wide tables in HBase. Wide tables have many columns, possibly thousands, typically
+            recording many attributes for an important subject such as a user of an online service. These tables
+            are also often sparse, that is, most of the columns values are <codeph>NULL</codeph>, 0,
+            <codeph>false</codeph>, empty string, or other blank or placeholder value. (For example, any particular
+            web site user might have never used some site feature, filled in a certain field in their profile,
+            visited a particular part of the site, and so on.) A typical query against this kind of table is to
+            look up a single row to retrieve all the information about a specific subject, rather than summing,
+            averaging, or filtering millions of rows as in typical Impala-managed tables.
+          </p>
+          <p>
+            Or the HBase table could be joined with a larger Impala-managed table. For example, analyze the large
+            Impala table representing web traffic for a site and pick out 50 users who view the most pages. Join
+            that result with the wide user table in HBase to look up attributes of those users. The HBase side of
+            the join would result in 50 efficient single-row lookups in HBase, rather than scanning the entire user
+            table.
+          </p>
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept audience="Cloudera" id="hbase_create_new">
+
+    <title>Creating a New HBase Table for Impala to Use</title>
+
+    <conbody>
+
+      <p>
+        You can create an HBase-backed table through a <codeph>CREATE TABLE</codeph> statement in the Hive shell,
+        without going into the HBase shell at all:
+      </p>
+
+      <!-- To do:
+        Add example. (Not critical because this subtopic is currently hidden.)
+      -->
+    </conbody>
+  </concept>
+
+  <concept audience="Cloudera" id="hbase_reuse_existing">
+
+    <title>Associate Impala with an Existing HBase Table</title>
+
+    <conbody>
+
+      <p>
+        If you already have some HBase tables created through the HBase shell, you can make them accessible to
+        Impala through a <codeph>CREATE TABLE</codeph> statement in the Hive shell:
+      </p>
+
+      <!-- To do:
+        Add example. (Not critical because this subtopic is currently hidden.)
+      -->
+    </conbody>
+  </concept>
+
+  <concept audience="Cloudera" id="hbase_column_families">
+
+    <title>Map HBase Columns and Column Families to Impala Columns</title>
+
+    <conbody>
+
+      <p/>
+    </conbody>
+  </concept>
+
+  <concept id="hbase_loading">
+
+    <title>Loading Data into an HBase Table</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="ETL"/>
+      <data name="Category" value="Ingest"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        The Impala <codeph>INSERT</codeph> statement works for HBase tables. The <codeph>INSERT ... VALUES</codeph>
+        syntax is ideally suited to HBase tables, because inserting a single row is an efficient operation for an
+        HBase table. (For regular Impala tables, with data files in HDFS, the tiny data files produced by
+        <codeph>INSERT ... VALUES</codeph> are extremely inefficient, so you would not use that technique with
+        tables containing any significant data volume.)
+      </p>
+
+      <!-- To do:
+        Add examples throughout this section.
+      -->
+
+      <p>
+        When you use the <codeph>INSERT ... SELECT</codeph> syntax, the result in the HBase table could be fewer
+        rows than you expect. HBase only stores the most recent version of each unique row key, so if an
+        <codeph>INSERT ... SELECT</codeph> statement copies over multiple rows containing the same value for the
+        key column, subsequent queries will only return one row with each key column value:
+      </p>
+
+      <p>
+        Although Impala does not have an <codeph>UPDATE</codeph> statement, you can achieve the same effect by
+        doing successive <codeph>INSERT</codeph> statements using the same value for the key column each time:
+      </p>
+
+    </conbody>
+  </concept>
+
+  <concept id="hbase_limitations">
+
+    <title>Limitations and Restrictions of the Impala and HBase Integration</title>
+
+    <conbody>
+
+      <p>
+        The Impala integration with HBase has the following limitations and restrictions, some inherited from the
+        integration between HBase and Hive, and some unique to Impala:
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            If you issue a <codeph>DROP TABLE</codeph> for an internal (Impala-managed) table that is mapped to an
+            HBase table, the underlying table is not removed in HBase. The Hive <codeph>DROP TABLE</codeph>
+            statement also removes the HBase table in this case.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            The <codeph>INSERT OVERWRITE</codeph> statement is not available for HBase tables. You can insert new
+            data, or modify an existing row by inserting a new row with the same key value, but not replace the
+            entire contents of the table. You can do an <codeph>INSERT OVERWRITE</codeph> in Hive if you need this
+            capability.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            If you issue a <codeph>CREATE TABLE LIKE</codeph> statement for a table mapped to an HBase table, the
+            new table is also an HBase table, but inherits the same underlying HBase table name as the original.
+            The new table is effectively an alias for the old one, not a new table with identical column structure.
+            Avoid using <codeph>CREATE TABLE LIKE</codeph> for HBase tables, to avoid any confusion.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Copying data into an HBase table using the Impala <codeph>INSERT ... SELECT</codeph> syntax might
+            produce fewer new rows than are in the query result set. If the result set contains multiple rows with
+            the same value for the key column, each row supercedes any previous rows with the same key value.
+            Because the order of the inserted rows is unpredictable, you cannot rely on this technique to preserve
+            the <q>latest</q> version of a particular key value.
+          </p>
+        </li>
+        <li rev="2.3.0">
+          <p>
+            Because the complex data types (<codeph>ARRAY</codeph>, <codeph>STRUCT</codeph>, and <codeph>MAP</codeph>)
+            available in <keyword keyref="impala23_full"/> and higher are currently only supported in Parquet tables, you cannot
+            use these types in HBase tables that are queried through Impala.
+          </p>
+        </li>
+        <li>
+          <p conref="../shared/impala_common.xml#common/hbase_no_load_data"/>
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="hbase_queries">
+
+    <title>Examples of Querying HBase Tables from Impala</title>
+
+    <conbody>
+
+      <p>
+        The following examples create an HBase table with four column families,
+        create a corresponding table through Hive,
+        then insert and query the table through Impala.
+      </p>
+      <p>
+        In HBase shell, the table
+        name is quoted in <codeph>CREATE</codeph> and <codeph>DROP</codeph> statements. Tables created in HBase
+        begin in <q>enabled</q> state; before dropping them through the HBase shell, you must issue a
+        <codeph>disable '<varname>table_name</varname>'</codeph> statement.
+      </p>
+
+<codeblock>$ hbase shell
+15/02/10 16:07:45
+HBase Shell; enter 'help&lt;RETURN>' for list of supported commands.
+Type "exit&lt;RETURN>" to leave the HBase Shell
+Version 0.94.2-cdh4.2.0, rUnknown, Fri Feb 15 11:51:18 PST 2013
+
+hbase(main):001:0> create 'hbasealltypessmall', 'boolsCF', 'intsCF', 'floatsCF', 'stringsCF'
+0 row(s) in 4.6520 seconds
+
+=> Hbase::Table - hbasealltypessmall
+hbase(main):006:0> quit
+</codeblock>
+
+        <p>
+          Issue the following <codeph>CREATE TABLE</codeph> statement in the Hive shell. (The Impala <codeph>CREATE
+          TABLE</codeph> statement currently does not support the <codeph>STORED BY</codeph> clause, so you switch into Hive to
+          create the table, then back to Impala and the <cmdname>impala-shell</cmdname> interpreter to issue the
+          queries.)
+        </p>
+
+        <p>
+          This example creates an external table mapped to the HBase table, usable by both Impala and Hive. It is
+          defined as an external table so that when dropped by Impala or Hive, the original HBase table is not touched at all.
+        </p>
+
+        <p>
+          The <codeph>WITH SERDEPROPERTIES</codeph> clause
+          specifies that the first column (<codeph>ID</codeph>) represents the row key, and maps the remaining
+          columns of the SQL table to HBase column families. The mapping relies on the ordinal order of the
+          columns in the table, not the column names in the <codeph>CREATE TABLE</codeph> statement.
+          The first column is defined to be the lookup key; the
+          <codeph>STRING</codeph> data type produces the fastest key-based lookups for HBase tables.
+        </p>
+
+        <note>
+          For Impala with HBase tables, the most important aspect to ensure good performance is to use a
+          <codeph>STRING</codeph> column as the row key, as shown in this example.
+        </note>
+
+<codeblock>$ hive
+Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j.properties
+Hive history file=/tmp/cloudera/hive_job_log_cloudera_201502101610_1980712808.txt
+hive> use hbase;
+OK
+Time taken: 4.095 seconds
+hive> CREATE EXTERNAL TABLE hbasestringids (
+    >   id string,
+    >   bool_col boolean,
+    >   tinyint_col tinyint,
+    >   smallint_col smallint,
+    >   int_col int,
+    >   bigint_col bigint,
+    >   float_col float,
+    >   double_col double,
+    >   date_string_col string,
+    >   string_col string,
+    >   timestamp_col timestamp)
+    > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
+    > WITH SERDEPROPERTIES (
+    >   "hbase.columns.mapping" =
+    >   ":key,boolsCF:bool_col,intsCF:tinyint_col,intsCF:smallint_col,intsCF:int_col,intsCF:\
+    >   bigint_col,floatsCF:float_col,floatsCF:double_col,stringsCF:date_string_col,\
+    >   stringsCF:string_col,stringsCF:timestamp_col"
+    > )
+    > TBLPROPERTIES("hbase.table.name" = "hbasealltypessmall");
+OK
+Time taken: 2.879 seconds
+hive> quit;
+</codeblock>
+
+        <p>
+          Once you have established the mapping to an HBase table, you can issue DML statements and queries
+          from Impala. The following example shows a series of <codeph>INSERT</codeph>
+          statements followed by a query.
+          The ideal kind of query from a performance standpoint
+          retrieves a row from the table based on a row key
+          mapped to a string column.
+          An initial <codeph>INVALIDATE METADATA <varname>table_name</varname></codeph>
+          statement makes the table created through Hive visible to Impala.
+        </p>
+
+<codeblock>$ impala-shell -i localhost -d hbase
+Starting Impala Shell without Kerberos authentication
+Connected to localhost:21000
+Server version: impalad version 2.1.0-cdh4 RELEASE (build d520a9cdea2fc97e8d5da9fbb0244e60ee416bfa)
+Welcome to the Impala shell. Press TAB twice to see a list of available commands.
+
+Copyright (c) 2012 Cloudera, Inc. All rights reserved.
+
+(Shell build version: Impala Shell v2.1.0-cdh4 (d520a9c) built on Mon Dec  8 21:41:17 PST 2014)
+Query: use `hbase`
+[localhost:21000] > invalidate metadata hbasestringids;
+Fetched 0 row(s) in 0.09s
+[localhost:21000] > desc hbasestringids;
++-----------------+-----------+---------+
+| name            | type      | comment |
++-----------------+-----------+---------+
+| id              | string    |         |
+| bool_col        | boolean   |         |
+| double_col      | double    |         |
+| float_col       | float     |         |
+| bigint_col      | bigint    |         |
+| int_col         | int       |         |
+| smallint_col    | smallint  |         |
+| tinyint_col     | tinyint   |         |
+| date_string_col | string    |         |
+| string_col      | string    |         |
+| timestamp_col   | timestamp |         |
++-----------------+-----------+---------+
+Fetched 11 row(s) in 0.02s
+[localhost:21000] > insert into hbasestringids values ('0001',true,3.141,9.94,1234567,32768,4000,76,'2014-12-31','Hello world',now());
+Inserted 1 row(s) in 0.26s
+[localhost:21000] > insert into hbasestringids values ('0002',false,2.004,6.196,1500,8000,129,127,'2014-01-01','Foo bar',now());
+Inserted 1 row(s) in 0.12s
+[localhost:21000] > select * from hbasestringids where id = '0001';
++------+----------+------------+-------------------+------------+---------+--------------+-------------+-----------------+-------------+-------------------------------+
+| id   | bool_col | double_col | float_col         | bigint_col | int_col | smallint_col | tinyint_col | date_string_col | string_col  | timestamp_col                 |
++------+----------+------------+-------------------+------------+---------+--------------+-------------+-----------------+-------------+-------------------------------+
+| 0001 | true     | 3.141      | 9.939999580383301 | 1234567    | 32768   | 4000         | 76          | 2014-12-31      | Hello world | 2015-02-10 16:36:59.764838000 |
++------+----------+------------+-------------------+------------+---------+--------------+-------------+-----------------+-------------+-------------------------------+
+Fetched 1 row(s) in 0.54s
+</codeblock>
+
+        <note conref="../shared/impala_common.xml#common/invalidate_metadata_hbase"/>
+<!--      </section> -->
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_hbase_cache_blocks.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_hbase_cache_blocks.xml b/docs/topics/impala_hbase_cache_blocks.xml
new file mode 100644
index 0000000..480b31c
--- /dev/null
+++ b/docs/topics/impala_hbase_cache_blocks.xml
@@ -0,0 +1,40 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="hbase_cache_blocks">
+
+  <title>HBASE_CACHE_BLOCKS Query Option</title>
+  <titlealts audience="PDF"><navtitle>HBASE_CACHE_BLOCKS</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="HBase"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">HBASE_CACHE_BLOCKS query option</indexterm>
+      Setting this option is equivalent to calling the
+        <codeph>setCacheBlocks</codeph> method of the class <xref
+        href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html"
+        scope="external" format="html"
+        >org.apache.hadoop.hbase.client.Scan</xref>, in an HBase Java
+      application. Helps to control the memory pressure on the HBase
+      RegionServer, in conjunction with the <codeph>HBASE_CACHING</codeph> query
+      option. </p>
+
+    <p conref="../shared/impala_common.xml#common/type_boolean"/>
+    <p conref="../shared/impala_common.xml#common/default_false_0"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_hbase.xml#impala_hbase"/>,
+      <xref href="impala_hbase_caching.xml#hbase_caching"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_hbase_caching.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_hbase_caching.xml b/docs/topics/impala_hbase_caching.xml
new file mode 100644
index 0000000..03755fd
--- /dev/null
+++ b/docs/topics/impala_hbase_caching.xml
@@ -0,0 +1,45 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="hbase_caching">
+
+  <title>HBASE_CACHING Query Option</title>
+  <titlealts audience="PDF"><navtitle>HBASE_CACHING</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="HBase"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">HBASE_CACHING query option</indexterm>
+      Setting this option is equivalent to calling the
+        <codeph>setCaching</codeph> method of the class <xref
+        href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html"
+        scope="external" format="html"
+        >org.apache.hadoop.hbase.client.Scan</xref>, in an HBase Java
+      application. Helps to control the memory pressure on the HBase
+      RegionServer, in conjunction with the <codeph>HBASE_CACHE_BLOCKS</codeph>
+      query option. </p>
+
+    <p>
+      <b>Type:</b> <codeph>BOOLEAN</codeph>
+    </p>
+
+    <p>
+      <b>Default:</b> 0
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_hbase.xml#impala_hbase"/>,
+      <xref href="impala_hbase_cache_blocks.xml#hbase_cache_blocks"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_hints.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_hints.xml b/docs/topics/impala_hints.xml
new file mode 100644
index 0000000..4c5145f
--- /dev/null
+++ b/docs/topics/impala_hints.xml
@@ -0,0 +1,249 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="hints">
+
+  <title>Query Hints in Impala SELECT Statements</title>
+  <titlealts audience="PDF"><navtitle>Hints</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">hints</indexterm>
+      The Impala SQL dialect supports query hints, for fine-tuning the inner workings of queries. Specify hints as
+      a temporary workaround for expensive queries, where missing statistics or other factors cause inefficient
+      performance.
+    </p>
+
+    <p>
+      Hints are most often used for the most resource-intensive kinds of Impala queries:
+    </p>
+
+    <ul>
+      <li>
+        Join queries involving large tables, where intermediate result sets are transmitted across the network to
+        evaluate the join conditions.
+      </li>
+
+      <li>
+        Inserting into partitioned Parquet tables, where many memory buffers could be allocated on each host to
+        hold intermediate results for each partition.
+      </li>
+    </ul>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p>
+      You can represent the hints as keywords surrounded by <codeph>[]</codeph> square brackets; include the
+      brackets in the text of the SQL statement.
+    </p>
+
+<codeblock>SELECT STRAIGHT_JOIN <varname>select_list</varname> FROM
+<varname>join_left_hand_table</varname>
+  JOIN [{BROADCAST|SHUFFLE}]
+<varname>join_right_hand_table</varname>
+<varname>remainder_of_query</varname>;
+
+INSERT <varname>insert_clauses</varname>
+  [{SHUFFLE|NOSHUFFLE}]
+  SELECT <varname>remainder_of_query</varname>;
+</codeblock>
+
+    <p rev="2.0.0">
+      In <keyword keyref="impala20_full"/> and higher, you can also specify the hints inside comments that use
+      either the <codeph>/* */</codeph> or <codeph>--</codeph> notation. Specify a <codeph>+</codeph> symbol
+      immediately before the hint name.
+    </p>
+
+<codeblock rev="2.0.0">SELECT STRAIGHT_JOIN <varname>select_list</varname> FROM
+<varname>join_left_hand_table</varname>
+  JOIN /* +BROADCAST|SHUFFLE */
+<varname>join_right_hand_table</varname>
+<varname>remainder_of_query</varname>;
+
+SELECT <varname>select_list</varname> FROM
+<varname>join_left_hand_table</varname>
+  JOIN -- +BROADCAST|SHUFFLE
+<varname>join_right_hand_table</varname>
+<varname>remainder_of_query</varname>;
+
+INSERT <varname>insert_clauses</varname>
+  /* +SHUFFLE|NOSHUFFLE */
+  SELECT <varname>remainder_of_query</varname>;
+
+INSERT <varname>insert_clauses</varname>
+  -- +SHUFFLE|NOSHUFFLE
+  SELECT <varname>remainder_of_query</varname>;
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      With both forms of hint syntax, include the <codeph>STRAIGHT_JOIN</codeph>
+      keyword immediately after the <codeph>SELECT</codeph> keyword to prevent Impala from
+      reordering the tables in a way that makes the hint ineffective.
+    </p>
+
+    <p>
+      To reduce the need to use hints, run the <codeph>COMPUTE STATS</codeph> statement against all tables involved
+      in joins, or used as the source tables for <codeph>INSERT ... SELECT</codeph> operations where the
+      destination is a partitioned Parquet table. Do this operation after loading data or making substantial
+      changes to the data within each table. Having up-to-date statistics helps Impala choose more efficient query
+      plans without the need for hinting. See <xref href="impala_perf_stats.xml#perf_stats"/> for details and
+      examples.
+    </p>
+
+    <p>
+      To see which join strategy is used for a particular query, examine the <codeph>EXPLAIN</codeph> output for
+      that query. See <xref href="impala_explain_plan.xml#perf_explain"/> for details and examples.
+    </p>
+
+    <p>
+      <b>Hints for join queries:</b>
+    </p>
+
+    <p>
+      The <codeph>[BROADCAST]</codeph> and <codeph>[SHUFFLE]</codeph> hints control the execution strategy for join
+      queries. Specify one of the following constructs immediately after the <codeph>JOIN</codeph> keyword in a
+      query:
+    </p>
+
+    <ul>
+      <li>
+        <codeph>[SHUFFLE]</codeph> - Makes that join operation use the <q>partitioned</q> technique, which divides
+        up corresponding rows from both tables using a hashing algorithm, sending subsets of the rows to other
+        nodes for processing. (The keyword <codeph>SHUFFLE</codeph> is used to indicate a <q>partitioned join</q>,
+        because that type of join is not related to <q>partitioned tables</q>.) Since the alternative
+        <q>broadcast</q> join mechanism is the default when table and index statistics are unavailable, you might
+        use this hint for queries where broadcast joins are unsuitable; typically, partitioned joins are more
+        efficient for joins between large tables of similar size.
+      </li>
+
+      <li>
+        <codeph>[BROADCAST]</codeph> - Makes that join operation use the <q>broadcast</q> technique that sends the
+        entire contents of the right-hand table to all nodes involved in processing the join. This is the default
+        mode of operation when table and index statistics are unavailable, so you would typically only need it if
+        stale metadata caused Impala to mistakenly choose a partitioned join operation. Typically, broadcast joins
+        are more efficient in cases where one table is much smaller than the other. (Put the smaller table on the
+        right side of the <codeph>JOIN</codeph> operator.)
+      </li>
+    </ul>
+
+    <p>
+      <b>Hints for INSERT ... SELECT queries:</b>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/insert_hints"/>
+
+    <p>
+      <b>Suggestions versus directives:</b>
+    </p>
+
+    <p>
+      In early Impala releases, hints were always obeyed and so acted more like directives. Once Impala gained join
+      order optimizations, sometimes join queries were automatically reordered in a way that made a hint
+      irrelevant. Therefore, the hints act more like suggestions in Impala 1.2.2 and higher.
+    </p>
+
+    <p>
+      To force Impala to follow the hinted execution mechanism for a join query, include the
+      <codeph>STRAIGHT_JOIN</codeph> keyword in the <codeph>SELECT</codeph> statement. See
+      <xref href="impala_perf_joins.xml#straight_join"/> for details. When you use this technique, Impala does not
+      reorder the joined tables at all, so you must be careful to arrange the join order to put the largest table
+      (or subquery result set) first, then the smallest, second smallest, third smallest, and so on. This ordering lets Impala do the
+      most I/O-intensive parts of the query using local reads on the DataNodes, and then reduce the size of the
+      intermediate result set as much as possible as each subsequent table or subquery result set is joined.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+    <p>
+      Queries that include subqueries in the <codeph>WHERE</codeph> clause can be rewritten internally as join
+      queries. Currently, you cannot apply hints to the joins produced by these types of queries.
+    </p>
+
+    <p>
+      Because hints can prevent queries from taking advantage of new metadata or improvements in query planning,
+      use them only when required to work around performance issues, and be prepared to remove them when they are
+      no longer required, such as after a new Impala release or bug fix.
+    </p>
+
+    <p>
+      In particular, the <codeph>[BROADCAST]</codeph> and <codeph>[SHUFFLE]</codeph> hints are expected to be
+      needed much less frequently in Impala 1.2.2 and higher, because the join order optimization feature in
+      combination with the <codeph>COMPUTE STATS</codeph> statement now automatically choose join order and join
+      mechanism without the need to rewrite the query and add hints. See
+      <xref href="impala_perf_joins.xml#perf_joins"/> for details.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/compatibility_blurb"/>
+
+    <p rev="2.0.0">
+      The hints embedded within <codeph>--</codeph> comments are compatible with Hive queries. The hints embedded
+      within <codeph>/* */</codeph> comments or <codeph>[ ]</codeph> square brackets are not recognized by or not
+      compatible with Hive. For example, Hive raises an error for Impala hints within <codeph>/* */</codeph>
+      comments because it does not recognize the Impala hint names.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/view_blurb"/>
+
+    <p rev="2.0.0">
+      If you use a hint in the query that defines a view, the hint is preserved when you query the view. Impala
+      internally rewrites all hints in views to use the <codeph>--</codeph> comment notation, so that Hive can
+      query such views without errors due to unrecognized hint names.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      For example, this query joins a large customer table with a small lookup table of less than 100 rows. The
+      right-hand table can be broadcast efficiently to all nodes involved in the join. Thus, you would use the
+      <codeph>[broadcast]</codeph> hint to force a broadcast join strategy:
+    </p>
+
+<codeblock>select straight_join customer.address, state_lookup.state_name
+  from customer join <b>[broadcast]</b> state_lookup
+  on customer.state_id = state_lookup.state_id;</codeblock>
+
+    <p>
+      This query joins two large tables of unpredictable size. You might benchmark the query with both kinds of
+      hints and find that it is more efficient to transmit portions of each table to other nodes for processing.
+      Thus, you would use the <codeph>[shuffle]</codeph> hint to force a partitioned join strategy:
+    </p>
+
+<codeblock>select straight_join weather.wind_velocity, geospatial.altitude
+  from weather join <b>[shuffle]</b> geospatial
+  on weather.lat = geospatial.lat and weather.long = geospatial.long;</codeblock>
+
+    <p>
+      For joins involving three or more tables, the hint applies to the tables on either side of that specific
+      <codeph>JOIN</codeph> keyword. The <codeph>STRAIGHT_JOIN</codeph> keyword ensures that joins are processed
+      in a predictable order from left to right. For example, this query joins
+      <codeph>t1</codeph> and <codeph>t2</codeph> using a partitioned join, then joins that result set to
+      <codeph>t3</codeph> using a broadcast join:
+    </p>
+
+<codeblock>select straight_join t1.name, t2.id, t3.price
+  from t1 join <b>[shuffle]</b> t2 join <b>[broadcast]</b> t3
+  on t1.id = t2.id and t2.id = t3.id;</codeblock>
+
+    <!-- To do: This is a good place to add more sample output showing before and after EXPLAIN plans. -->
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      For more background information about join queries, see <xref href="impala_joins.xml#joins"/>. For
+      performance considerations, see <xref href="impala_perf_joins.xml#perf_joins"/>.
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_howto_rm.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_howto_rm.xml b/docs/topics/impala_howto_rm.xml
new file mode 100644
index 0000000..2a7494b
--- /dev/null
+++ b/docs/topics/impala_howto_rm.xml
@@ -0,0 +1,420 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="howto_impala_rm">
+
+ <title>How to Configure Resource Management for Impala</title>
+ <prolog>
+  <metadata>
+   <data name="Category" value="Impala"/>
+   <data name="Category" value="Admission Control"/>
+   <data name="Category" value="Resource Management"/>
+  </metadata>
+ </prolog>
+ <conbody>
+  <p>Impala includes features that balance and maximize resources in your CDH cluster. This topic describes
+  how you can enhance a CDH cluster using Impala to improve efficiency.</p>
+  <p outputclass="toc inpage">A typical deployment uses the following.</p>
+  <ul audience="HTML">
+   <li audience="HTML">Creating Static Service Pools</li>
+   <li audience="HTML">Using Admission Control <ul>
+     <li audience="HTML">Setting Per-query Memory Limits</li>
+     <li audience="HTML">Creating Dynamic Resource Pools</li>
+    </ul></li>
+
+  </ul>
+ </conbody>
+ <concept id="static_service_pools">
+   <title>Creating Static Service Pools</title>
+  <conbody>
+  <p>Use Static Service Pools to allocate dedicated resources for Impala and other services to allow
+    for predictable resource availability. </p>
+    <p>Static service pools isolate services from one another, so that high load on one service has
+    bounded impact on other services. You can use Cloudera Manager to configure static service pools
+    that control memory, CPU and Disk I/O.</p>
+    <p >The following screenshot shows a sample configuration for Static Service Pools in
+    Cloudera Manager:</p>
+   <draft-comment author="ddawson">
+    <p>Need accurate numbers - or can we remove services other than HDFS, Impala, and YARN? Matt
+     Jacobs is going to run these numbers by someone in the field.</p>
+   </draft-comment>
+   <p>
+    <image href="../images/howto_static_server_pools_config.png" placement="break"
+     id="PDF_33" align="center" scale="33" audience="HTML"/>
+    <image href="../images/howto_static_server_pools_config.png" placement="break"
+     id="HTML_SCALEFIT" align="center" scalefit="yes" audience="PDF"/>
+   </p>
+   <p>
+    <ul id="ul_tkw_4rs_pw">
+     <li >
+      <p >HDFS always needs to have a minimum of 5-10% of the resources.</p>
+     </li>
+     <li >
+      Generally, YARN and Impala split the rest of the resources.
+     
+     <ul id="ul_ukw_4rs_pw">
+      <li >
+       <p >For mostly batch workloads, you might allocate YARN 60%, Impala 30%, and HDFS
+        10%. </p>
+      </li>
+      <li >
+       <p >For mostly ad hoc query workloads, you might allocate Impala 60%, YARN 30%, and
+        HDFS 10%.</p>
+      </li>
+     </ul></li>
+    </ul>
+   </p>
+   </conbody>
+  </concept>
+ <concept id="enable_admission_control">
+  <title>Using Admission Control</title>
+  <conbody>
+   <p>Within the constraints of the static service pool, you can further subdivide Impala's
+    resources using Admission Control. You configure Impala Admission Control pools in the Cloudera
+    Manager Dynamic Resource Pools page.</p>
+   <p>You use Admission Control to divide usage between Dynamic Resource Pools in multitenant use
+    cases. Allocating resources judiciously allows your most important queries to run faster and
+    more reliably.</p>
+   <p>
+    <note>In this context, Impala Dynamic Resource Pools are different than the default YARN Dynamic
+     Resource Pools. You can turn on Dynamic Resource Pools that are exclusively for use by
+     Impala.</note>
+   </p>
+   <p>Admission Control is enabled by default.</p>
+   <p>A Dynamic Resource Pool has the following properties:<ul id="ul_blk_jjg_sw">
+     <li><b>Max Running Queries</b>: Maximum number of concerrently executing queries in the pool
+      before incoming queries are queued.</li>
+     <li><b>Max Memory Resources</b>: Maximum memory used by queries in the pool before incoming
+      queries are queued. This value is used at the time of admission and is not enforced at query
+      runtime.</li>
+     <li><b>Default Query Memory Limit</b>: Defines the maximum amount of memory a query can
+      allocate on each node. This is enforced at runtime. If the query attempts to use more memory,
+      it is forced to spill, if possible. Otherwise, it is cancelled. The total memory that can be
+      used by a query is the <codeph>MEM_LIMIT</codeph> times the number of nodes.</li>
+     <li><b>Max Queued Queries</b>: Maximum number of queries that can be queued in the pool before
+      additional queries are rejected.</li>
+     <li><b>Queue Timeout</b>: Specifies how long queries can wait in the queue before they are
+      cancelled with a timeout error.</li>
+    </ul></p>
+  </conbody>
+ </concept>
+ <concept id="set_per_query_memory_limits">
+  <title>Setting Per-query Memory Limits</title>
+  <conbody>
+   <p>Use per-query memory limits to prevent queries from consuming excessive memory resources that
+    impact other queries. <ph rev="upstream">Cloudera</ph> recommends that you set the query memory limits whenever
+    possible.</p>
+   <p>If you set the <b>Pool Max Mem Resources</b> for a resource pool, Impala attempts to throttle
+    queries if there is not enough memory to run them within the specified resources.</p>
+   <p>Only use admission control with maximum memory resources if you can ensure there are query
+    memory limits. Set the pool <b>Default Query Memory Limit</b> to be certain. You can override
+    this setting with the <codeph>query</codeph> option, if necessary.</p>
+   <p>Typically, you set query memory limits using the <codeph>set MEM_LIMIT=Xg;</codeph> query
+    option. When you find the right value for your business case, memory-based admission control
+    works well. The potential downside is that queries that attempt to use more memory might perform
+    poorly or even be cancelled.</p>
+   <p>To find a reasonable default query memory limit:<ol id="ol_ydt_xhy_pw">
+     <li>Run the workload.</li>
+     <li>In Cloudera Manager, go to <menucascade>
+       <uicontrol>Impala</uicontrol>
+       <uicontrol>Queries</uicontrol>
+      </menucascade>.</li>
+     <li>Click <uicontrol>Select Attributes</uicontrol>.</li>
+     <li>Select <uicontrol>Per Node Peak Memory Usage</uicontrol> and click
+       <uicontrol>Update</uicontrol>.</li>
+     <li>Allow the system time to gather information, then click the <uicontrol>Show
+       Histogram</uicontrol> icon to see the results.<image placement="break"
+       href="../images/howto_show_histogram.png" align="center" id="image_hmv_xky_pw"/></li>
+     <li>Use the histogram to find a value that accounts for most queries. Queries that require more
+      resources than this limit should explicitly set the memory limit to ensure they can run to
+       completion.<draft-comment author="ddawson">This chart uses bad sample data - we will change
+       the chart when we have real numbers from the sample use case.</draft-comment><image
+       placement="break" href="../images/howto_per_node_peak_memory_usage.png" align="center"
+       id="image_ehn_hly_pw" scalefit="yes"/></li>
+    </ol></p>
+  </conbody>
+ </concept>
+ <concept id="concept_en4_3sy_pw">
+  <title>Creating Dynamic Resource Pools</title>
+  <conbody>
+   <p>A dynamic resource pool is a named configuration of resources and a policy for scheduling the
+    resources among Impala queries running in the pool. Dynamic resource pools allow you to schedule
+    and allocate resources to Impala queries based on a user's access to specific pools and the
+    resources available to those pools. </p>
+   <p>This example creates both production and development resource pools or queues. It assumes you
+    have 3 worker nodes with 24GiB of RAM each for an aggregate memory of 72000MiB. This pool
+    configuration allocates the Production queue twice the memory resources of the Development
+    queue, and a higher number of concurrent queries.</p>
+   <p>To create a Production dynamic resource pool for Impala:</p>
+    <ol>
+     <li>In Cloudera Manager, select <menucascade>
+      <uicontrol>Clusters</uicontrol>
+      <uicontrol>Dynamic Resource Pool Configuration</uicontrol>
+     </menucascade>.</li>
+     <li>Click the <uicontrol>Impala Admission Control</uicontrol> tab.</li>
+     <li>Click <b>Create Resource Pool</b>.</li>
+     <li>Specify a name and resource limits for the Production pool:<ul id="ul_rjt_wqv_2v">
+       <li>In the <b>Resource Pool Name</b> field, enter <userinput>Production</userinput>.</li>
+       <li>In the <uicontrol>Max Memory</uicontrol> field, enter <userinput>48000</userinput>.</li>
+      <li>In the <uicontrol>Default Query Memory Limit</uicontrol> field, enter
+        <userinput>1600</userinput>.</li>
+       <li>In the <uicontrol>Max Running Queries</uicontrol> field, enter
+       <userinput>10</userinput>.</li>
+       <li>In the <uicontrol>Max Queued Queries</uicontrol> field, enter
+       <userinput>200</userinput>.</li>
+      </ul></li>
+     <li>Click <uicontrol>Create</uicontrol>.</li>
+     <li>Click <uicontrol>Refresh Dynamic Resource Pools</uicontrol>.</li>
+    </ol>
+    <p>The Production queue runs up to 10 queries at once. If the total memory requested
+    by these queries exceeds 48000 MiB, it holds the next query in the queue until the memory is
+    released. It also prevents a query from running if it needs more memory than is currently
+    available. Admission Control holds the next query if either Max Running Queries is reached, or
+    the pool Max Memory limit is reached.</p>
+   <p>Here, Max Memory resources and Default Query Memory Limit throttle throughput to 10 queries,
+    so setting Max Running Queries might not be necessary, though it does not hurt to do so. Most
+    users set Max Running Queries when they cannot pick good numbers for memory. Since users can
+    override the query option <varname>mem_limit</varname>, setting the Max Running Queries property
+    might make sense.</p>
+    <p>To create a Development dynamic resource pool for Impala:</p>
+
+    <ol>
+     <li>In Cloudera Manager, select <menucascade>
+      <uicontrol>Clusters</uicontrol>
+      <uicontrol>Dynamic Resource Pool Configuration</uicontrol>
+     </menucascade>.</li>
+     <li>Click the <uicontrol>Impala Admission Control</uicontrol> tab.</li>
+     <li>Click <b>Create Resource Pool</b>.</li>
+     <li>Specify a name and resource limits for the Development pool:<ul id="ul_j42_q3z_pw">
+       <li>In the <b>Resource Pool Name</b> field, enter <userinput>Development</userinput>.</li>
+       <li>In the <uicontrol>Max Memory</uicontrol> field, enter <userinput>24000</userinput>.</li>
+      <li>In the <uicontrol>Default Query Memory Limit</uicontrol> field, enter
+        <userinput>8000</userinput>.</li>
+       <li>In the <uicontrol>Max Running Queries</uicontrol> field, enter 1.</li>
+       <li>In the <uicontrol>Max Queued Queries</uicontrol> field, enter 100.</li>
+      </ul></li>
+     <li>Click <uicontrol>Create</uicontrol>.</li>
+     <li>Click <uicontrol>Refresh Dynamic Resource Pools</uicontrol>.<p>The Development queue runs
+      one query at a time. If the total memory required by the query exceeds 24000 MiB, it holds the
+      query until memory is released.</p></li>
+    </ol>
+  </conbody>
+  <concept id="setting_placement_rules">
+   <title>Understanding Placement Rules</title>
+   <conbody>
+    <p>Placement rules determine how queries are mapped to resource pools. The standard settings are
+     to use a specified pool when specified; otherwise, use the default pool.</p>
+    <p>For example, you can use the SET statement to select the pool in which to run a
+     query.<codeblock>SET REQUEST_POOL=Production;</codeblock></p>
+    <p>If you do not use a <codeph>SET</codeph> statement, queries are run in the default pool.</p>
+   </conbody>
+  </concept>
+  <concept id="setting_access_control_on_pools">
+  <title>Setting Access Control on Pools</title>
+  <conbody>
+   <p>You can specify that only cetain users and groups are allowed to use the pools you define.</p>
+   <p>To create a Development dynamic resource pool for Impala:</p>
+   <ol>
+    <li>In Cloudera Manager, select <menucascade>
+       <uicontrol>Clusters</uicontrol>
+       <uicontrol>Dynamic Resource Pool Configuration</uicontrol>
+      </menucascade>.</li>
+    <li>Click the <uicontrol>Impala Admission Control</uicontrol> tab.</li>
+    <li>Click the <uicontrol>Edit</uicontrol> button for the Production pool.</li>
+    <li>Click the Submission Access Control tab.</li>
+    <li>Select <uicontrol>Allow these users and groups to submit to this pool</uicontrol>.</li>
+    <li>Enter a comma-separated list of users who can use the pool.
+     <image placement="break"
+      href="../images/howto_access_control.png" align="center" scalefit="yes"/>
+     </li>
+    
+    <li>Click <uicontrol>Save</uicontrol>.</li>
+   </ol>
+  </conbody>
+ </concept>
+ </concept>
+<concept id="impala_resource_management_example">
+ <title>Impala Resource Management Example</title>
+ <conbody>
+   <p>Anne Chang is administrator for an enterprise data hub that runs a number of workloads,
+    including Impala. </p>
+   <p>Anne has a 20-node cluster that uses Cloudera Manager static partitioning. Because of the
+    heavy Impala workload, Anne needs to make sure Impala gets enough resources. While the best
+    configuration values might not be known in advance, she decides to start by allocating 50% of
+    resources to Impala. Each node has 128 GiB dedicated to each impalad. Impala has 2560 GiB in
+    aggregate that can be shared across the resource pools she creates.</p>
+   <p>Next, Anne studies the workload in more detail. After some research, she might choose to
+    revisit these initial values for static partitioning. </p>
+   <p>To figure out how to further allocate Impala\u2019s resources, Anne needs to consider the workloads
+    and users, and determine their requirements. There are a few main sources of Impala queries: <ul
+     id="ul_ml3_sf2_5w">
+     <li>Large reporting queries executed by an external process/tool. These are critical business
+      intelligence queries that are important for business decisions. It is important that they get
+      the resources they need to run. There typically are not many of these queries at a given
+      time.</li>
+     <li>Frequent, small queries generated by a web UI. These queries scan a limited amount of data
+      and do not require expensive joins or aggregations. These queries are important, but not as
+      critical, perhaps the client tries resending the query or the end user refreshes the
+      page.</li>
+     <li>Occasionally, expert users might run ad-hoc queries. The queries can vary significantly in
+      their resource requirements. While Anne wants a good experience for these users, it is hard to
+      control what they do (for example, submitting inefficient or incorrect queries by mistake).
+      Anne restricts these queries by default and tells users to reach out to her if they need more
+      resources. </li>
+    </ul></p>
+   <p>To set up admission control for this workload, Anne first runs the workloads independently, so
+    that she can observe the workload\u2019s resource usage in Cloudera Manager. If they could not easily
+    be run manually, but had been run in the past, Anne uses the history information from Cloudera
+    Manager. It can be helpful to use other search criteria (for example, <i>user</i>) to isolate
+    queries by workload. Anne uses the Cloudera Manager chart for Per-Node Peak Memory usage to
+    identify the maximum memory requirements for the queries. </p>
+   <p>From this data, Anne observes the following about the queries in the groups above:<ul
+     id="ul_amq_ng2_5w">
+     <li> Large reporting queries use up to 32 GiB per node. There are typically 1 or 2 queries
+      running at a time. On one occasion, she observed that 3 of these queries were running
+      concurrently. Queries can take 3 minutes to complete.</li>
+     <li>Web UI-generated queries use between 100 MiB per node to usually less than 4 GiB per node
+      of memory, but occasionally as much as 10 GiB per node. Queries take, on average, 5 seconds,
+      and there can be as many as 140 incoming queries per minute.</li>
+     <li>Anne has little data on ad hoc queries, but some are trivial (approximately 100 MiB per
+      node), others join several tables (requiring a few GiB per node), and one user submitted a
+      huge cross join of all tables that used all system resources (that was likely a mistake).</li>
+    </ul></p>
+   <p>Based on these observations, Anne creates the admission control configuration with the
+    following pools: </p>
+   <section id="section_yjc_h32_5w">
+    <title>XL_Reporting</title>
+    <p>
+     <table frame="all" rowsep="1" colsep="1" id="XL_Reporting_Table">
+      <tgroup cols="2">
+       <colspec colname="c1" colnum="1" colwidth="1.0*"/>
+       <colspec colname="c2" colnum="2" colwidth="1.0*"/>
+       <thead>
+        <row>
+         <entry>Property</entry>
+         <entry>Value</entry>
+        </row>
+       </thead>
+       <tbody>
+        <row>
+         <entry>Max Memory</entry>
+         <entry>1280 GiB</entry>
+        </row>
+        <row>
+         <entry>Default Query Memory Limit</entry>
+         <entry>32 GiB</entry>
+        </row>
+        <row>
+         <entry>Max Running Queries</entry>
+         <entry>2</entry>
+        </row>
+        <row>
+         <entry>Queue Timeout</entry>
+         <entry>5 minutes</entry>
+        </row>
+       </tbody>
+      </tgroup>
+     </table>
+    </p>
+    <p>This pool is for large reporting queries. In order to support running 2 queries at a time,
+     the pool memory resources are set to 1280 GiB (aggregate cluster memory). This is for 2
+     queries, each with 32 GiB per node, across 20 nodes. Anne sets the pool\u2019s Default Query Memory
+     Limit to 32 GiB so that no query uses more than 32 GiB on any given node. She sets Max Running
+     Queries to 2 (though it is not necessary she do so). She increases the pool\u2019s queue timeout to
+     5 minutes in case a third query comes in and has to wait. She does not expect more than 3
+     concurrent queries, and she does not want them to wait that long anyway, so she does not
+     increase the queue timeout. If the workload increases in the future, she might choose to adjust
+     the configuration or buy more hardware. </p>
+   </section>
+   <section id="section_xm3_j32_5w"><title>HighThroughput_UI</title>
+    <p>
+     <table frame="all" rowsep="1" colsep="1" id="High_Throughput_UI_Table">
+      <tgroup cols="2">
+       <colspec colname="c1" colnum="1" colwidth="1.0*"/>
+       <colspec colname="c2" colnum="2" colwidth="1.0*"/>
+       <thead>
+        <row>
+         <entry>Property</entry>
+         <entry>Value</entry>
+        </row>
+       </thead>
+       <tbody>
+        <row>
+         <entry>Max Memory</entry>
+         <entry>960 GiB (inferred)</entry>
+        </row>
+        <row>
+         <entry>Default Query Memory Limit</entry>
+         <entry>4 GiB</entry>
+        </row>
+        <row>
+         <entry>Max Running Queries</entry>
+         <entry>12</entry>
+        </row>
+        <row>
+         <entry>Queue Timeout</entry>
+         <entry>5 minutes</entry>
+        </row>
+       </tbody>
+      </tgroup>
+     </table>
+    </p>
+    <p>This pool is used for the small, high throughput queries generated by the web tool. Anne sets
+     the Default Query Memory Limit to 4 GiB per node, and sets Max Running Queries to 12. This
+     implies a maximum amount of memory per node used by the queries in this pool: 48 GiB per node
+     (12 queries * 4 GiB per node memory limit).</p><p>Notice that Anne does not set the pool memory resources, but does set the pool\u2019s Default Query
+     Memory Limit. This is intentional: admission control processes queries faster when a pool uses
+     the Max Running Queries limit instead of the peak memory resources.</p><p>This should be enough memory for most queries, since only a few go over 4 GiB per node. For those
+     that do require more memory, they can probably still complete with less memory (spilling if
+     necessary). If, on occasion, a query cannot run with this much memory and it fails, Anne might
+     reconsider this configuration later, or perhaps she does not need to worry about a few rare
+     failures from this web UI.</p><p>With regard to throughput, since these queries take around 5 seconds and she is allowing 12
+     concurrent queries, the pool should be able to handle approximately 144 queries per minute,
+     which is enough for the peak maximum expected of 140 queries per minute. In case there is a
+     large burst of queries, Anne wants them to queue. The default maximum size of the queue is
+     already 200, which should be more than large enough. Anne does not need to change it.</p></section>
+   <section id="section_asm_yj2_5w"><title>Default</title>
+    <p>
+     <table frame="all" rowsep="1" colsep="1" id="default_table">
+      <tgroup cols="2">
+       <colspec colname="c1" colnum="1" colwidth="1.0*"/>
+       <colspec colname="c2" colnum="2" colwidth="1.0*"/>
+       <thead>
+        <row>
+         <entry>Property</entry>
+         <entry>Value</entry>
+        </row>
+       </thead>
+       <tbody>
+        <row>
+         <entry>Max Memory</entry>
+         <entry>320 GiB</entry>
+        </row>
+        <row>
+         <entry>Default Query Memory Limit</entry>
+         <entry>4 GiB</entry>
+        </row>
+        <row>
+         <entry>Max Running Queries</entry>
+         <entry>Unlimited</entry>
+        </row>
+        <row>
+         <entry>Queue Timeout</entry>
+         <entry>60 Seconds</entry>
+        </row>
+       </tbody>
+      </tgroup>
+     </table>
+    </p><p>The default pool (which already exists) is a catch all for ad-hoc queries. Anne wants to use the
+     remaining memory not used by the first two pools, 16 GiB per node (XL_Reporting uses 64 GiB per
+     node, High_Throughput_UI uses 48 GiB per node). For the other pools to get the resources they
+     expect, she must still set the Max Memory resources and the Default Query Memory Limit. She
+     sets the Max Memory resources to 320 GiB (16 * 20). She sets the Default Query Memory Limit to
+     4 GiB per node for now. That is somewhat arbitrary, but satisfies some of the ad hoc queries
+     she observed. If someone writes a bad query by mistake, she does not actually want it using all
+     the system resources. If a user has a large query to submit, an expert user can override the
+     Default Query Memory Limit (up to 16 GiB per node, since that is bound by the pool Max Memory
+     resources). If that is still insufficient for this user\u2019s workload, the user should work with
+     Anne to adjust the settings and perhaps create a dedicated pool for the workload.</p></section>
+  </conbody>
+</concept>
+</concept>


[07/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_shell_options.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_shell_options.xml b/docs/topics/impala_shell_options.xml
new file mode 100644
index 0000000..7da27af
--- /dev/null
+++ b/docs/topics/impala_shell_options.xml
@@ -0,0 +1,580 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="shell_options">
+
+  <title>impala-shell Configuration Options</title>
+  <titlealts audience="PDF"><navtitle>Configuration Options</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Configuring"/>
+      <data name="Category" value="impala-shell"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      You can specify the following options when starting the <codeph>impala-shell</codeph> command to change how
+      shell commands are executed. The table shows the format to use when specifying each option on the command
+      line, or through the <filepath>$HOME/.impalarc</filepath> configuration file.
+    </p>
+
+    <note>
+      <p>
+        These options are different than the configuration options for the <codeph>impalad</codeph> daemon itself.
+        For the <codeph>impalad</codeph> options, see <xref href="impala_config_options.xml#config_options"/>.
+      </p>
+    </note>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="shell_option_summary">
+
+    <title>Summary of impala-shell Configuration Options</title>
+
+    <conbody>
+
+      <p>
+        The following table shows the names and allowed arguments for the <cmdname>impala-shell</cmdname>
+        configuration options. You can specify options on the command line, or in a configuration file as described
+        in <xref href="impala_shell_options.xml#shell_config_file"/>.
+      </p>
+
+      <table>
+        <tgroup cols="3">
+          <colspec colname="1" colwidth="10*"/>
+          <colspec colname="2" colwidth="10*"/>
+          <colspec colname="3" colwidth="20*"/>
+          <thead>
+            <row>
+              <entry>
+                Command-Line Option
+              </entry>
+              <entry rev="2.0.0">
+                Configuration File Setting
+              </entry>
+              <entry>
+                Explanation
+              </entry>
+            </row>
+          </thead>
+          <tbody>
+            <row>
+              <entry>
+                <p>
+                  -B or --delimited
+                </p>
+              </entry>
+              <entry rev="2.0.0">
+                <p>
+                  write_delimited=true
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Causes all query results to be printed in plain format as a delimited text file. Useful for
+                  producing data files to be used with other Hadoop components. Also useful for avoiding the
+                  performance overhead of pretty-printing all output, especially when running benchmark tests using
+                  queries returning large result sets. Specify the delimiter character with the
+                  <codeph>--output_delimiter</codeph> option. Store all query results in a file rather than
+                  printing to the screen with the <codeph>-B</codeph> option. Added in Impala 1.0.1.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  --print_header
+                </p>
+              </entry>
+              <entry rev="2.0.0">
+                <p>
+                  print_header=true
+                </p>
+              </entry>
+              <entry>
+                <p/>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  -o <varname>filename</varname> or --output_file <varname>filename</varname>
+                </p>
+              </entry>
+              <entry rev="2.0.0">
+                <p>
+                  output_file=<varname>filename</varname>
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Stores all query results in the specified file. Typically used to store the results of a single
+                  query issued from the command line with the <codeph>-q</codeph> option. Also works for
+                  interactive sessions; you see the messages such as number of rows fetched, but not the actual
+                  result set. To suppress these incidental messages when combining the <codeph>-q</codeph> and
+                  <codeph>-o</codeph> options, redirect <codeph>stderr</codeph> to <codeph>/dev/null</codeph>.
+                  Added in Impala 1.0.1.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  --output_delimiter=<varname>character</varname>
+                </p>
+              </entry>
+              <entry rev="2.0.0">
+                <p>
+                  output_delimiter=<varname>character</varname>
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Specifies the character to use as a delimiter between fields when query results are printed in
+                  plain format by the <codeph>-B</codeph> option. Defaults to tab (<codeph>'\t'</codeph>). If an
+                  output value contains the delimiter character, that field is quoted, escaped by doubling quotation marks, or both. Added in
+                  Impala 1.0.1.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  -p or --show_profiles
+                </p>
+              </entry>
+              <entry rev="2.0.0">
+                <p>
+                  show_profiles=true
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Displays the query execution plan (same output as the <codeph>EXPLAIN</codeph> statement) and a
+                  more detailed low-level breakdown of execution steps, for every query executed by the shell.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  -h or --help
+                </p>
+              </entry>
+              <entry rev="2.0.0">
+                <p>
+                  N/A
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Displays help information.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  -i <varname>hostname</varname> or
+                  --impalad=<varname>hostname</varname>[:<varname>portnum</varname>]
+                </p>
+              </entry>
+              <entry rev="2.0.0">
+                <p>
+                  impalad=<varname>hostname</varname>[:<varname>portnum</varname>]
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Connects to the <codeph>impalad</codeph> daemon on the specified host. The default port of 21000
+                  is assumed unless you provide another value. You can connect to any host in your cluster that is
+                  running <codeph>impalad</codeph>. If you connect to an instance of <codeph>impalad</codeph> that
+                  was started with an alternate port specified by the <codeph>--fe_port</codeph> flag, provide that
+                  alternative port.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  -q <varname>query</varname> or --query=<varname>query</varname>
+                </p>
+              </entry>
+              <entry rev="2.0.0">
+                <p>
+                  query=<varname>query</varname>
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Passes a query or other <cmdname>impala-shell</cmdname> command from the command line. The
+                  <cmdname>impala-shell</cmdname> interpreter immediately exits after processing the statement. It
+                  is limited to a single statement, which could be a <codeph>SELECT</codeph>, <codeph>CREATE
+                  TABLE</codeph>, <codeph>SHOW TABLES</codeph>, or any other statement recognized in
+                  <codeph>impala-shell</codeph>. Because you cannot pass a <codeph>USE</codeph> statement and
+                  another query, fully qualify the names for any tables outside the <codeph>default</codeph>
+                  database. (Or use the <codeph>-f</codeph> option to pass a file with a <codeph>USE</codeph>
+                  statement followed by other queries.)
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  -f <varname>query_file</varname> or --query_file=<varname>query_file</varname>
+                </p>
+              </entry>
+              <entry rev="2.0.0">
+                <p>
+                  query_file=<varname>path_to_query_file</varname>
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Passes a SQL query from a file. Multiple statements must be semicolon (;) delimited.
+                  <ph rev="2.3.0">In <keyword keyref="impala23_full"/> and higher, you can specify a filename of <codeph>-</codeph>
+                  to represent standard input. This feature makes it convenient to use <cmdname>impala-shell</cmdname>
+                  as part of a Unix pipeline where SQL statements are generated dynamically by other tools.</ph>
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  -k or --kerberos
+                </p>
+              </entry>
+              <entry rev="2.0.0">
+                <p>
+                  use_kerberos=true
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Kerberos authentication is used when the shell connects to <codeph>impalad</codeph>. If Kerberos
+                  is not enabled on the instance of <codeph>impalad</codeph> to which you are connecting, errors
+                  are displayed.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  -s <varname>kerberos_service_name</varname> or --kerberos_service_name=<varname>name</varname>
+                </p>
+              </entry>
+              <entry rev="2.0.0">
+                <p>
+                  kerberos_service_name=<varname>name</varname>
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Instructs <codeph>impala-shell</codeph> to authenticate to a particular <codeph>impalad</codeph>
+                  service principal. If a <varname>kerberos_service_name</varname> is not specified,
+                  <codeph>impala</codeph> is used by default. If this option is used in conjunction with a
+                  connection in which Kerberos is not supported, errors are returned.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  -V or --verbose
+                </p>
+              </entry>
+              <entry rev="2.0.0">
+                <p>
+                  verbose=true
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Enables verbose output.
+                </p>
+              </entry>
+            </row>
+            <row>
+<!-- Confirm verbose=true/false really is the same as verbose vs. quiet. -->
+              <entry>
+                <p>
+                  --quiet
+                </p>
+              </entry>
+              <entry rev="2.0.0">
+                <p>
+                  verbose=false
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Disables verbose output.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  -v or --version
+                </p>
+              </entry>
+              <entry rev="2.0.0">
+                <p>
+                  version=true
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Displays version information.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  -c
+                </p>
+              </entry>
+              <entry rev="2.0.0">
+                <p>
+                  ignore_query_failure=true
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Continues on query failure.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  -r or --refresh_after_connect
+                </p>
+              </entry>
+              <entry rev="2.0.0">
+                <p>
+                  refresh_after_connect=true
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Updates Impala metadata upon connection. Same as running the
+                  <codeph><xref href="impala_invalidate_metadata.xml#invalidate_metadata">INVALIDATE
+                  METADATA</xref></codeph> statement after connecting. (This option was originally named when the
+                  <codeph>REFRESH</codeph> statement did the extensive metadata updates now performed by
+                  <codeph>INVALIDATE METADATA</codeph>.)
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  -d <varname>default_db</varname> or --database=<varname>default_db</varname>
+                </p>
+              </entry>
+              <entry rev="2.0.0">
+                <p>
+                  default_db=<varname>default_db</varname>
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Specifies the database to be used on startup. Same as running the
+                  <codeph><xref href="impala_use.xml#use">USE</xref></codeph> statement after connecting. If not
+                  specified, a database named <codeph>DEFAULT</codeph> is used.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                -ssl
+              </entry>
+              <entry rev="2.0.0">
+                ssl=true
+              </entry>
+              <entry>
+                Enables TLS/SSL for <cmdname>impala-shell</cmdname>.
+              </entry>
+            </row>
+            <row>
+              <entry>
+                --ca_cert=<varname>path_to_certificate</varname>
+              </entry>
+              <entry rev="2.0.0">
+                ca_cert=<varname>path_to_certificate</varname>
+              </entry>
+              <entry>
+                The local pathname pointing to the third-party CA certificate, or to a copy of the server
+                certificate for self-signed server certificates. If <codeph>--ca_cert</codeph> is not set,
+                <cmdname>impala-shell</cmdname> enables TLS/SSL, but does not validate the server certificate. This is
+                useful for connecting to a known-good Impala that is only running over TLS/SSL, when a copy of the
+                certificate is not available (such as when debugging customer installations).
+              </entry>
+            </row>
+            <row rev="1.2.2">
+              <entry>
+                -l
+              </entry>
+              <entry rev="2.0.0">
+                use_ldap=true
+              </entry>
+              <entry>
+                Enables LDAP authentication.
+              </entry>
+            </row>
+            <row rev="1.2.2">
+              <entry>
+                -u
+              </entry>
+              <entry rev="2.0.0">
+                user=<varname>user_name</varname>
+              </entry>
+              <entry>
+                Supplies the username, when LDAP authentication is enabled by the <codeph>-l</codeph> option.
+                (Specify the short username, not the full LDAP distinguished name.) The shell then prompts
+                interactively for the password.
+              </entry>
+            </row>
+            <row rev="2.5.0 IMPALA-1934">
+              <entry>
+                --ldap_password_cmd=<varname>command</varname>
+              </entry>
+              <entry>
+                N/A
+              </entry>
+              <entry>
+                Specifies a command to run to retrieve the LDAP password,
+                when LDAP authentication is enabled by the <codeph>-l</codeph> option.
+                If the command includes space-separated arguments, enclose the command and
+                its arguments in quotation marks.
+              </entry>
+            </row>
+            <row rev="2.0.0">
+              <entry>
+                --config_file=<varname>path_to_config_file</varname>
+              </entry>
+              <entry>
+                N/A
+              </entry>
+              <entry>
+                Specifies the path of the file containing <cmdname>impala-shell</cmdname> configuration settings.
+                The default is <filepath>$HOME/.impalarc</filepath>. This setting can only be specified on the
+                command line.
+              </entry>
+            </row>
+            <row rev="2.3.0">
+              <entry>--live_progress</entry>
+              <entry>N/A</entry>
+              <entry>Prints a progress bar showing roughly the percentage complete for each query.
+              The information is updated interactively as the query progresses.
+              See <xref href="impala_live_progress.xml#live_progress"/>.</entry>
+            </row>
+            <row rev="2.3.0">
+              <entry>--live_summary</entry>
+              <entry>N/A</entry>
+              <entry>Prints a detailed report, similar to the <codeph>SUMMARY</codeph> command, showing progress details for each phase of query execution.
+              The information is updated interactively as the query progresses.
+              See <xref href="impala_live_summary.xml#live_summary"/>.</entry>
+            </row>
+            <row rev="2.5.0 IMPALA-1079">
+              <entry>--var=<varname>variable_name</varname>=<varname>value</varname></entry>
+              <entry>N/A</entry>
+              <entry>
+                Defines a substitution variable that can be used within the <cmdname>impala-shell</cmdname> session.
+                The variable can be substituted into statements processed by the <codeph>-q</codeph> or <codeph>-f</codeph> options,
+                or in an interactive shell session.
+                Within a SQL statement, you substitute the value by using the notation <codeph>${var:<varname>variable_name</varname>}</codeph>.
+                This feature is available in <keyword keyref="impala25_full"/> and higher.
+              </entry>
+            </row>
+          </tbody>
+        </tgroup>
+      </table>
+    </conbody>
+  </concept>
+
+  <concept id="shell_config_file">
+
+    <title>impala-shell Configuration File</title>
+
+    <conbody>
+
+      <p>
+        You can define a set of default options for your <cmdname>impala-shell</cmdname> environment, stored in the
+        file <filepath>$HOME/.impalarc</filepath>. This file consists of key-value pairs, one option per line.
+        Everything after a <codeph>#</codeph> character on a line is treated as a comment and ignored.
+      </p>
+
+      <p>
+        The configuration file must contain a header label <codeph>[impala]</codeph>, followed by the options
+        specific to <cmdname>impala-shell</cmdname>. (This standard convention for configuration files lets you
+        use a single file to hold configuration options for multiple applications.)
+      </p>
+
+      <p>
+        To specify a different filename or path for the configuration file, specify the argument
+        <codeph>--config_file=<varname>path_to_config_file</varname></codeph> on the
+        <cmdname>impala-shell</cmdname> command line.
+      </p>
+
+      <p>
+        The names of the options in the configuration file are similar (although not necessarily identical) to the
+        long-form command-line arguments to the <cmdname>impala-shell</cmdname> command. For the names to use, see
+        <xref href="impala_shell_options.xml#shell_option_summary"/>.
+      </p>
+
+      <p>
+        Any options you specify on the <cmdname>impala-shell</cmdname> command line override any corresponding
+        options within the configuration file.
+      </p>
+
+      <p>
+        The following example shows a configuration file that you might use during benchmarking tests. It sets
+        verbose mode, so that the output from each SQL query is followed by timing information.
+        <cmdname>impala-shell</cmdname> starts inside the database containing the tables with the benchmark data,
+        avoiding the need to issue a <codeph>USE</codeph> statement or use fully qualified table names.
+      </p>
+
+      <p>
+        In this example, the query output is formatted as delimited text rather than enclosed in ASCII art boxes,
+        and is stored in a file rather than printed to the screen. Those options are appropriate for benchmark
+        situations, so that the overhead of <cmdname>impala-shell</cmdname> formatting and printing the result set
+        does not factor into the timing measurements. It also enables the <codeph>show_profiles</codeph> option.
+        That option prints detailed performance information after each query, which might be valuable in
+        understanding the performance of benchmark queries.
+      </p>
+
+<codeblock>[impala]
+verbose=true
+default_db=tpc_benchmarking
+write_delimited=true
+output_delimiter=,
+output_file=/home/tester1/benchmark_results.csv
+show_profiles=true
+</codeblock>
+
+      <p>
+        The following example shows a configuration file that connects to a specific remote Impala node, runs a
+        single query within a particular database, then exits. You would typically use this kind of single-purpose
+        configuration setting with the <cmdname>impala-shell</cmdname> command-line option
+        <codeph>--config_file=<varname>path_to_config_file</varname></codeph>, to easily select between many
+        predefined queries that could be run against different databases, hosts, or even different clusters. To run
+        a sequence of statements instead of a single query, specify the configuration option
+        <codeph>query_file=<varname>path_to_query_file</varname></codeph> instead.
+      </p>
+
+<codeblock>[impala]
+impalad=impala-test-node1.example.com
+default_db=site_stats
+# Issue a predefined query and immediately exit.
+query=select count(*) from web_traffic where event_date = trunc(now(),'dd')
+</codeblock>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_shell_running_commands.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_shell_running_commands.xml b/docs/topics/impala_shell_running_commands.xml
new file mode 100644
index 0000000..70b4593
--- /dev/null
+++ b/docs/topics/impala_shell_running_commands.xml
@@ -0,0 +1,265 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="shell_running_commands">
+
+  <title>Running Commands and SQL Statements in impala-shell</title>
+  <titlealts audience="PDF"><navtitle>Running Commands and SQL Statements</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="impala-shell"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      For information on available commands, see
+      <xref href="impala_shell_commands.xml#shell_commands"/>. You can see the full set of available
+      commands by pressing TAB twice, for example:
+    </p>
+
+<codeblock>[impalad-host:21000] &gt;
+connect   describe  explain   help      history   insert    quit      refresh   select    set       shell     show      use       version
+[impalad-host:21000] &gt;</codeblock>
+
+    <note>
+      Commands must be terminated by a semi-colon. A command can span multiple lines.
+    </note>
+
+    <p>
+      For example:
+    </p>
+
+<codeblock>[localhost:21000] &gt; select *
+                  &gt; from t1
+                  &gt; limit 5;
++---------+-----------+
+| s1      | s2        |
++---------+-----------+
+| hello   | world     |
+| goodbye | cleveland |
++---------+-----------+
+</codeblock>
+
+    <p>
+      A comment is considered part of the statement it precedes, so when you enter a <codeph>--</codeph> or
+      <codeph>/* */</codeph> comment, you get a continuation prompt until you finish entering a statement ending
+      with a semicolon:
+    </p>
+
+<codeblock>[localhost:21000] &gt; -- This is a test comment
+                  &gt; show tables like 't*';
++--------+
+| name   |
++--------+
+| t1     |
+| t2     |
+| tab1   |
+| tab2   |
+| tab3   |
+| text_t |
++--------+
+</codeblock>
+
+    <p>
+      Use the up-arrow and down-arrow keys to cycle through and edit previous commands.
+      <cmdname>impala-shell</cmdname> uses the <codeph>readline</codeph> library and so supports a standard set of
+      keyboard shortcuts for editing and cursor movement, such as <codeph>Ctrl-A</codeph> for beginning of line and
+      <codeph>Ctrl-E</codeph> for end of line.
+    </p>
+
+    <p rev="2.5.0 IMPALA-2179 IMPALA-2180">
+      In <keyword keyref="impala25_full"/> and higher, you can define substitution variables to be used within SQL statements
+      processed by <cmdname>impala-shell</cmdname>. On the command line, you specify the option
+      <codeph>--var=<varname>variable_name</varname>=<varname>value</varname></codeph>.
+      Within an interactive session or a script file processed by the <codeph>-f</codeph> option, you specify
+      a <codeph>SET</codeph> command using the notation <codeph>SET VAR:<varname>variable_name</varname>=<varname>value</varname></codeph>.
+      Within a SQL statement, you substitute the value by using the notation <codeph>${var:<varname>variable_name</varname>}</codeph>.
+    </p>
+
+    <note>
+      Because this feature is part of <cmdname>impala-shell</cmdname> rather than the <cmdname>impalad</cmdname>
+      backend, make sure the client system you are connecting from has the most recent <cmdname>impala-shell</cmdname>.
+      You can use this feature with a new <cmdname>impala-shell</cmdname> connecting to an older <cmdname>impalad</cmdname>,
+      but not the reverse.
+    </note>
+
+    <p rev="2.5.0 IMPALA-2179 IMPALA-2180">
+      For example, here are some <cmdname>impala-shell</cmdname> commands that define substitution variables and then
+      use them in SQL statements executed through the <codeph>-q</codeph> and <codeph>-f</codeph> options.
+      Notice how the <codeph>-q</codeph> argument strings are single-quoted to prevent shell expansion of the
+      <codeph>${var:value}</codeph> notation, and any string literals within the queries are enclosed by double quotation marks.
+    </p>
+
+<codeblock rev="2.5.0 IMPALA-2179 IMPALA-2180">
+$ impala-shell --var=tname=table1 --var=colname=x --var=coltype=string -q 'create table ${var:tname} (${var:colname} ${var:coltype}) stored as parquet'
+Starting Impala Shell without Kerberos authentication
+Connected to <varname>hostname</varname>
+Server version: <varname>impalad_version</varname>
+Query: create table table1 (x string) stored as parquet
+
+$ NEW_STRING="hello world"
+$ impala-shell --var=tname=table1 --var=insert_val="$NEW_STRING" -q 'insert into ${var:tname} values ("${var:insert_val}")'
+Starting Impala Shell without Kerberos authentication
+Connected to <varname>hostname</varname>
+Server version: <varname>impalad_version</varname>
+Query: insert into table1 values ("hello world")
+Inserted 1 row(s) in 1.40s
+
+$ for VAL in foo bar bletch
+do
+  impala-shell --var=tname=table1 --var=insert_val="$VAL" -q 'insert into ${var:tname} values ("${var:insert_val}")'
+done
+...
+Query: insert into table1 values ("foo")
+Inserted 1 row(s) in 0.22s
+Query: insert into table1 values ("bar")
+Inserted 1 row(s) in 0.11s
+Query: insert into table1 values ("bletch")
+Inserted 1 row(s) in 0.21s
+
+$ echo "Search for what substring?" ; read answer
+Search for what substring?
+b
+$ impala-shell --var=tname=table1 -q 'select x from ${var:tname} where x like "%${var:answer}%"'
+Starting Impala Shell without Kerberos authentication
+Connected to <varname>hostname</varname>
+Server version: <varname>impalad_version</varname>
+Query: select x from table1 where x like "%b%"
++--------+
+| x      |
++--------+
+| bletch |
+| bar    |
++--------+
+Fetched 2 row(s) in 0.83s
+</codeblock>
+
+    <p rev="2.5.0 IMPALA-2179 IMPALA-2180">
+      Here is a substitution variable passed in by the <codeph>--var</codeph> option,
+      and then referenced by statements issued interactively. Then the variable is
+      cleared with the <codeph>UNSET</codeph> command, and defined again with the
+      <codeph>SET</codeph> command.
+    </p>
+
+<codeblock rev="2.5.0 IMPALA-2179 IMPALA-2180">
+$ impala-shell --quiet --var=tname=table1
+Starting Impala Shell without Kerberos authentication
+***********************************************************************************
+<varname>banner_message</varname>
+***********************************************************************************
+[<varname>hostname</varname>:21000] > select count(*) from ${var:tname};
++----------+
+| count(*) |
++----------+
+| 4        |
++----------+
+[<varname>hostname</varname>:21000] > unset var:tname;
+Unsetting variable TNAME
+[<varname>hostname</varname>:21000] > select count(*) from ${var:tname};
+Error: Unknown variable TNAME
+[<varname>hostname</varname>:21000] > set var:tname=table1;
+[<varname>hostname</varname>:21000] > select count(*) from ${var:tname};
++----------+
+| count(*) |
++----------+
+| 4        |
++----------+
+</codeblock>
+
+    <p rev="IMPALA-3397 CDH-40097">
+      The following example shows how the <codeph>SOURCE</codeph> command can execute
+      a series of statements from a file:
+    </p>
+
+<codeblock rev="IMPALA-3397 CDH-40097">
+$ cat commands.sql
+show databases;
+show tables in default;
+show functions in _impala_builtins like '*minute*';
+
+$ impala-shell -i localhost
+...
+[localhost:21000] > source commands.sql;
+Query: show databases
++------------------+----------------------------------------------+
+| name             | comment                                      |
++------------------+----------------------------------------------+
+| _impala_builtins | System database for Impala builtin functions |
+| default          | Default Hive database                        |
++------------------+----------------------------------------------+
+Fetched 2 row(s) in 0.06s
+Query: show tables in default
++-----------+
+| name      |
++-----------+
+| customers |
+| sample_07 |
+| sample_08 |
+| web_logs  |
++-----------+
+Fetched 4 row(s) in 0.02s
+Query: show functions in _impala_builtins like '*minute*'
++-------------+--------------------------------+-------------+---------------+
+| return type | signature                      | binary type | is persistent |
++-------------+--------------------------------+-------------+---------------+
+| INT         | minute(TIMESTAMP)              | BUILTIN     | true          |
+| TIMESTAMP   | minutes_add(TIMESTAMP, BIGINT) | BUILTIN     | true          |
+| TIMESTAMP   | minutes_add(TIMESTAMP, INT)    | BUILTIN     | true          |
+| TIMESTAMP   | minutes_sub(TIMESTAMP, BIGINT) | BUILTIN     | true          |
+| TIMESTAMP   | minutes_sub(TIMESTAMP, INT)    | BUILTIN     | true          |
++-------------+--------------------------------+-------------+---------------+
+Fetched 5 row(s) in 0.03s
+</codeblock>
+
+    <p rev="IMPALA-3397 CDH-40097">
+      The following example shows how a file that is run by the <codeph>SOURCE</codeph> command,
+      or through the <codeph>-q</codeph> or <codeph>-f</codeph> options of <cmdname>impala-shell</cmdname>,
+      can contain additional <codeph>SOURCE</codeph> commands.
+      The first file, <filepath>nested1.sql</filepath>, runs an <cmdname>impala-shell</cmdname> command
+      and then also runs the commands from <filepath>nested2.sql</filepath>.
+      This ability for scripts to call each other is often useful for code that sets up schemas for applications
+      or test environments.
+    </p>
+
+<codeblock rev="IMPALA-3397 CDH-40097">
+$ cat nested1.sql
+show functions in _impala_builtins like '*minute*';
+source nested2.sql
+$ cat nested2.sql
+show functions in _impala_builtins like '*hour*'
+
+$ impala-shell -i localhost -f nested1.sql
+Starting Impala Shell without Kerberos authentication
+Connected to localhost:21000
+...
+Query: show functions in _impala_builtins like '*minute*'
++-------------+--------------------------------+-------------+---------------+
+| return type | signature                      | binary type | is persistent |
++-------------+--------------------------------+-------------+---------------+
+| INT         | minute(TIMESTAMP)              | BUILTIN     | true          |
+| TIMESTAMP   | minutes_add(TIMESTAMP, BIGINT) | BUILTIN     | true          |
+| TIMESTAMP   | minutes_add(TIMESTAMP, INT)    | BUILTIN     | true          |
+| TIMESTAMP   | minutes_sub(TIMESTAMP, BIGINT) | BUILTIN     | true          |
+| TIMESTAMP   | minutes_sub(TIMESTAMP, INT)    | BUILTIN     | true          |
++-------------+--------------------------------+-------------+---------------+
+Fetched 5 row(s) in 0.01s
+Query: show functions in _impala_builtins like '*hour*'
++-------------+------------------------------+-------------+---------------+
+| return type | signature                    | binary type | is persistent |
++-------------+------------------------------+-------------+---------------+
+| INT         | hour(TIMESTAMP)              | BUILTIN     | true          |
+| TIMESTAMP   | hours_add(TIMESTAMP, BIGINT) | BUILTIN     | true          |
+| TIMESTAMP   | hours_add(TIMESTAMP, INT)    | BUILTIN     | true          |
+| TIMESTAMP   | hours_sub(TIMESTAMP, BIGINT) | BUILTIN     | true          |
+| TIMESTAMP   | hours_sub(TIMESTAMP, INT)    | BUILTIN     | true          |
++-------------+------------------------------+-------------+---------------+
+Fetched 5 row(s) in 0.01s
+</codeblock>
+
+  </conbody>
+</concept>


[30/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_faq.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_faq.xml b/docs/topics/impala_faq.xml
new file mode 100644
index 0000000..852beaf
--- /dev/null
+++ b/docs/topics/impala_faq.xml
@@ -0,0 +1,1877 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="faq">
+
+  <title>Impala Frequently Asked Questions</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="FAQs"/>
+      <data name="Category" value="Planning"/>
+      <data name="Category" value="Getting Started"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Here are the categories of frequently asked questions for Impala, the interactive SQL engine included with CDH.
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="faq_eval">
+
+    <title>Trying Impala</title>
+
+    <conbody>
+
+      <p outputclass="toc inpage" audience="PDF">
+        FAQs in this category:
+      </p>
+
+      <section id="faq_tryout">
+
+        <title>How do I try Impala out?</title>
+
+        <sectiondiv id="faq_try_impala">
+
+          <p>
+            To look at the core features and functionality on Impala, the easiest way to try out Impala is to
+            download the Cloudera QuickStart VM and start the Impala service through Cloudera Manager, then use
+            <cmdname>impala-shell</cmdname> in a terminal window or the Impala Query UI in the Hue web interface.
+          </p>
+
+          <p>
+            To do performance testing and try out the management features for Impala on a cluster, you need to move
+            beyond the QuickStart VM with its virtualized single-node environment. Ideally, download the Cloudera
+            Manager software to set up the cluster, then install the Impala software through Cloudera Manager.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_demo_vm">
+
+        <title>Does Cloudera offer a VM for demonstrating Impala?</title>
+
+        <sectiondiv id="faq_demo_vm_sect">
+
+          <p>
+            Cloudera offers a demonstration VM called the QuickStart VM, available in VMWare, VirtualBox, and KVM
+            formats. For more information, see
+<!-- Was:          <xref href="cloudera-content/cloudera-docs/DemoVMs/Cloudera-QuickStart-VM/cloudera_impala.html" scope="external" format="html">Cloudera Impala Demo VM</xref> -->
+<!-- Then was:          <xref href="cloudera-content/cloudera-docs/DemoVMs/Cloudera-QuickStart-VM/cloudera_quickstart_vm.html" scope="external" format="html">the Cloudera QuickStart VM</xref>. -->
+<!-- Finally(?) was:            <xref href="https://ccp.cloudera.com/display/SUPPORT/Cloudera+QuickStart+VM" scope="external" format="html">the Cloudera QuickStart VM</xref>. -->
+            <xref href="http://www.cloudera.com/content/support/en/downloads/quickstart_vms.html" scope="external" format="html">the
+            Cloudera QuickStart VM</xref>. After booting the QuickStart VM, many services are turned off by
+            default; in the Cloudera Manager UI that appears automatically, turn on Impala and any other components
+            that you want to try out.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_docs">
+
+        <title>Where can I find Impala documentation?</title>
+
+        <sectiondiv id="faq_doc">
+
+          <p>
+            Starting with Impala 1.3.0, Impala documentation is integrated with the CDH 5 documentation, in
+            addition to the standalone Impala documentation for use with CDH 4. For CDH 5, the core Impala
+            developer and administrator information remains in the associated
+<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/Impala/impala.html -->
+            <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/impala.html" scope="external" format="html">Impala
+            documentation</xref> portion. Information about Impala release notes, installation, configuration,
+            startup, and security is embedded in the corresponding CDH 5 guides.
+          </p>
+
+<!-- Same list is in impala.xml and Impala FAQs. Conref in both places. -->
+
+          <ul>
+            <li>
+              <xref href="impala_new_features.xml#new_features">New features</xref>
+            </li>
+
+            <li>
+              <xref href="impala_known_issues.xml#known_issues">Known and fixed issues</xref>
+            </li>
+
+            <li>
+              <xref href="impala_incompatible_changes.xml#incompatible_changes">Incompatible changes</xref>
+            </li>
+
+            <li>
+              <xref href="impala_install.xml#install">Installing Impala</xref>
+            </li>
+
+            <li>
+              <xref href="impala_upgrading.xml#upgrading">Upgrading Impala</xref>
+            </li>
+
+            <li>
+              <xref href="impala_config.xml#config">Configuring Impala</xref>
+            </li>
+
+            <li>
+              <xref href="impala_processes.xml#processes">Starting Impala</xref>
+            </li>
+
+            <li>
+              <xref href="impala_security.xml#security">Security for Impala</xref>
+            </li>
+
+            <li>
+<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH-Version-and-Packaging-Information/CDH-Version-and-Packaging-Information.html -->
+              <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/rg_vd.html" scope="external" format="html">CDH
+              Version and Packaging Information</xref>
+            </li>
+          </ul>
+
+          <p>
+            Information about the latest CDH 4-compatible Impala release remains at the
+<!-- Original URL: updated this from a /v1/ URL. -->
+            <xref href="http://www.cloudera.com/content/cloudera/en/documentation/impala/latest.html" scope="external" format="html">Impala
+            for CDH 4 Documentation</xref> page.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_more_info">
+
+        <title>Where can I get more information about Impala?</title>
+
+        <sectiondiv id="faq_more_info_sect">
+
+          <!-- JDR: Not changing these instances of 'Cloudera Impala' because those are the real titles of those books or blog posts. -->
+          <p>
+            More product information is available here:
+          </p>
+
+          <ul>
+            <li>
+              O'Reilly introductory e-book:
+              <xref href="http://radar.oreilly.com/2013/10/cloudera-impala-bringing-the-sql-and-hadoop-worlds-together.html" scope="external" format="html">Cloudera
+              Impala: Bringing the SQL and Hadoop Worlds Together</xref>
+            </li>
+
+            <li>
+              O'Reilly getting started guide for developers:
+              <xref href="http://shop.oreilly.com/product/0636920033936.do" scope="external" format="html">Getting
+              Started with Impala: Interactive SQL for Apache Hadoop</xref>
+            </li>
+
+            <li>
+              Blog:
+              <xref href="http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-queries-in-apache-hadoop-for-real" scope="external" format="html">Cloudera
+              Impala: Real-Time Queries in Apache Hadoop, For Real</xref>
+            </li>
+
+            <li>
+              Webinar:
+              <xref href="http://www.cloudera.com/content/cloudera/en/resources/library/recordedwebinar/impala-real-time-queries-in-hadoop-webinar-slides.html" scope="external" format="html">Introduction
+              to Impala</xref>
+            </li>
+
+            <li>
+              Product website page:
+              <xref href="http://www.cloudera.com/content/cloudera/en/products-and-services/cdh/impala.html" scope="external" format="html">Cloudera
+              Enterprise RTQ</xref>
+            </li>
+          </ul>
+
+          <p>
+            To see the latest release announcements for Impala, see the
+            <xref href="http://community.cloudera.com/t5/Release-Announcements/bd-p/RelAnnounce" scope="external" format="html">Cloudera
+            Announcements</xref> forum.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_community">
+
+        <title>How can I ask questions and provide feedback about Impala?</title>
+
+        <sectiondiv id="faq_qanda">
+
+          <ul>
+            <li>
+              Join the
+              <xref href="http://community.cloudera.com/t5/Interactive-Short-cycle-SQL/bd-p/Impala" scope="external" format="html">Impala
+              discussion forum</xref> and the
+              <xref href="https://groups.google.com/a/cloudera.org/forum/?fromgroups#!forum/impala-user" scope="external" format="html">Impala
+              mailing list</xref> to ask questions and provide feedback.
+            </li>
+
+            <li>
+              Use the <xref href="https://issues.cloudera.org/browse/IMPALA" scope="external" format="html">Impala
+              Jira project</xref> to log bug reports and requests for features.
+            </li>
+          </ul>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_tpcds">
+
+        <title>Where can I get sample data to try?</title>
+
+        <p>
+          You can get scripts that produce data files and set up an environment for TPC-DS style benchmark tests
+          from <xref href="https://github.com/cloudera/impala-tpcds-kit" scope="external" format="html">this Github
+          repository</xref>. In addition to being useful for experimenting with performance, the tables are suited
+          to experimenting with many aspects of SQL on Impala: they contain a good mixture of data types, data
+          distributions, partitioning, and relational data suitable for join queries.
+        </p>
+      </section>
+    </conbody>
+  </concept>
+
+  <concept id="faq_prereq">
+
+    <title>Impala System Requirements</title>
+  <prolog>
+    <metadata>
+      <!-- Normally I don't categorize subtopics under FAQs. Making an exception to beef up the EC2 category,
+           and to judge whether it makes sense to relax that rule a bit. -->
+      <data name="Category" value="Amazon"/>
+      <data name="Category" value="EC2"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p outputclass="toc inpage" audience="PDF">
+        FAQs in this category:
+      </p>
+
+      <section id="faq_prereqs">
+
+        <title>What are the software and hardware requirements for running Impala?</title>
+
+        <sectiondiv id="faq_system_reqs">
+
+          <p>
+            For information on Impala requirements, see <xref href="impala_prereqs.xml#prereqs"/>. Note that there
+            is often a minimum required level of Cloudera Manager for any given Impala version.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_memory_prereq">
+
+        <title>How much memory is required?</title>
+
+        <sectiondiv id="faq_mem_req">
+
+          <!-- To do:
+            Prefer to have more examples / citations for larger memory sizes. What are the most
+            memory-intensive operations that require or benefit from large mem size?
+            Actually that info should go into impala_scalability.xml and be xref'ed from here.
+          -->
+
+          <p>
+            Although Impala is not an in-memory database, when dealing with large tables and large result sets, you
+            should expect to dedicate a substantial portion of physical memory for the <cmdname>impalad</cmdname>
+            daemon. Recommended physical memory for an Impala node is 128 GB or higher. If practical, devote
+            approximately 80% of physical memory to Impala.
+<!-- The machines we typically run on have approximately 32-48 GB. -->
+          </p>
+
+          <p>
+            The amount of memory required for an Impala operation depends on several factors:
+          </p>
+
+          <ul>
+            <li>
+              <p>
+                The file format of the table. Different file formats represent the same data in more or fewer data
+                files. The compression and encoding for each file format might require a different amount of
+                temporary memory to decompress the data for analysis.
+              </p>
+            </li>
+
+            <li>
+              <p>
+                Whether the operation is a <codeph>SELECT</codeph> or an <codeph>INSERT</codeph>. For example,
+                Parquet tables require relatively little memory to query, because Impala reads and decompresses
+                data in 8 MB chunks. Inserting into a Parquet table is a more memory-intensive operation because
+                the data for each data file (potentially <ph rev="parquet_block_size">hundreds of megabytes,
+                depending on the value of the <codeph>PARQUET_FILE_SIZE</codeph> query option</ph>) is stored in
+                memory until encoded, compressed, and written to disk.
+<!-- In 2.0, default might be smaller than maximum. -->
+              </p>
+            </li>
+
+            <li>
+              <p>
+                Whether the table is partitioned or not, and whether a query against a partitioned table can take
+                advantage of partition pruning.
+              </p>
+            </li>
+
+            <li>
+              <p>
+                Whether the final result set is sorted by the <codeph>ORDER BY</codeph> clause.
+<!--
+<ph rev="obwl">Remember, Impala requires that all <codeph>ORDER BY</codeph> queries include a
+<codeph>LIMIT</codeph> clause too, either in the query syntax or implicitly
+through the <codeph>DEFAULT_ORDER_BY_LIMIT</codeph> query option.</ph>
+-->
+                Each Impala node scans and filters a portion of the total data, and applies the
+                <codeph>LIMIT</codeph> to its own portion of the result set. <ph rev="1.4.0">In Impala 1.4.0 and
+                higher, if the sort operation requires more memory than is available on any particular host, Impala
+                uses a temporary disk work area to perform the sort.</ph> The intermediate result sets
+<!-- (each with a maximum size of <codeph>LIMIT</codeph> rows) -->
+                are all sent back to the coordinator node, which does the final sorting and then applies the
+                <codeph>LIMIT</codeph> clause to the final result set.
+              </p>
+              <p>
+                For example, if you execute the query:
+<codeblock>select * from giant_table order by some_column limit 1000;</codeblock>
+                and your cluster has 50 nodes, then each of those 50 nodes will transmit a maximum of 1000 rows
+                back to the coordinator node. The coordinator node needs enough memory to sort
+                (<codeph>LIMIT</codeph> * <varname>cluster_size</varname>) rows, although in the end the final
+                result set is at most <codeph>LIMIT</codeph> rows, 1000 in this case.
+              </p>
+              <p>
+                Likewise, if you execute the query:
+<codeblock>select * from giant_table where test_val &gt; 100 order by some_column;</codeblock>
+                then each node filters out a set of rows matching the <codeph>WHERE</codeph> conditions, sorts the
+                results (with no size limit), and sends the sorted intermediate rows back to the coordinator node.
+                The coordinator node might need substantial memory to sort the final result set, and so might use a
+                temporary disk work area for that final phase of the query.
+              </p>
+            </li>
+
+            <li>
+              <p>
+                Whether the query contains any join clauses, <codeph>GROUP BY</codeph> clauses, analytic functions,
+                or <codeph>DISTINCT</codeph> operators. These operations all require some in-memory work areas that
+                vary depending on the volume and distribution of data. In Impala 2.0 and later, these kinds of
+                operations utilize temporary disk work areas if memory usage grows too large to handle. See
+                <xref href="impala_scalability.xml#spill_to_disk"/> for details.
+              </p>
+            </li>
+
+            <li>
+              <p>
+                The size of the result set. When intermediate results are being passed around between nodes, the
+                amount of data depends on the number of columns returned by the query. For example, it is more
+                memory-efficient to query only the columns that are actually needed in the result set rather than
+                always issuing <codeph>SELECT *</codeph>.
+              </p>
+            </li>
+
+            <li>
+              <p>
+                The mechanism by which work is divided for a join query. You use the <codeph>COMPUTE STATS</codeph>
+                statement, and query hints in the most difficult cases, to help Impala pick the most efficient
+                execution plan. See <xref href="impala_perf_joins.xml#perf_joins"/> for details.
+              </p>
+            </li>
+          </ul>
+
+          <p>
+            See <xref href="impala_prereqs.xml#prereqs_hardware"/> for more details and recommendations about
+            Impala hardware prerequisites.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_cpu_prereq">
+
+        <title>What processor type and speed does Cloudera recommend?</title>
+
+        <sectiondiv id="faq_cpu_req">
+
+          <p rev="CDH-24874">
+            Impala makes use of SSE 4.1 instructions.
+<!-- Commenting out of caution after IMPALA-160 and CDH-20937.
+            For best performance, use Nehalem or later for
+            Intel chips and Bulldozer or later for AMD chips.
+          Impala runs on older machines with the SSE3 instruction set,
+          but will not achieve the best performance.
+          -->
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_prereq_ec2">
+
+        <title>What EC2 instances are recommended for Impala?</title>
+
+        <p>
+          For large storage capacity and large I/O bandwidth, consider the <codeph>hs1.8xlarge</codeph> and
+          <codeph>cc2.8xlarge</codeph> instance types. Impala I/O patterns typically do not benefit enough from SSD
+          storage to make up for the lower overall size. For performance and security considerations for deploying
+          CDH and its components on AWS, see
+          <xref href="http://www.cloudera.com/content/dam/cloudera/Resources/PDF/whitepaper/AWS_Reference_Architecture_Whitepaper.pdf" scope="external" format="html">Cloudera
+          Enterprise Reference Architecture for AWS Deployments</xref>.
+        </p>
+      </section>
+    </conbody>
+  </concept>
+
+  <concept id="faq_features">
+
+    <title>Supported and Unsupported Functionality In Impala</title>
+
+    <conbody>
+
+      <p outputclass="toc inpage" audience="PDF">
+        FAQs in this category:
+      </p>
+
+      <section id="features">
+
+        <title>What are the main features of Impala?</title>
+
+        <sectiondiv id="faq_features_sql">
+
+          <ul>
+            <li>
+              A large set of SQL statements, including <xref href="impala_select.xml#select">SELECT</xref> and
+              <xref href="impala_insert.xml#insert">INSERT</xref>, with
+              <xref href="impala_joins.xml#joins">joins</xref>, <xref href="impala_subqueries.xml#subqueries"/>,
+              and <xref href="impala_analytic_functions.xml#analytic_functions"/>. Highly compatible with HiveQL,
+              and also including some vendor extensions. For more information, see
+              <xref href="impala_langref.xml#langref"/>.
+            </li>
+
+            <li>
+              Distributed, high-performance queries. See <xref href="impala_performance.xml#performance"/> for
+              information about Impala performance optimizations and tuning techniques for queries.
+            </li>
+
+            <li>
+              Using Cloudera Manager, you can deploy and manage your Impala services. Cloudera Manager is the best
+              way to get started with Impala on your cluster.
+            </li>
+
+            <li>
+              Using Hue for queries.
+            </li>
+
+            <li>
+              Appending and inserting data into tables through the
+              <xref href="impala_insert.xml#insert">INSERT</xref> statement. See
+              <xref href="impala_file_formats.xml#file_formats"/> for the details about which operations are
+              supported for which file formats.
+            </li>
+
+            <li>
+              ODBC: Impala is certified to run against MicroStrategy and Tableau, with restrictions. For more
+              information, see <xref href="impala_odbc.xml#impala_odbc"/>.
+            </li>
+
+            <li>
+              Querying data stored in HDFS and HBase in a single query. See
+              <xref href="impala_hbase.xml#impala_hbase"/> for details.
+            </li>
+
+            <li rev="2.2.0">
+              In Impala 2.2.0 and higher, querying data stored in the Amazon Simple Storage Service (S3). See
+              <xref href="impala_s3.xml#s3"/> for details.
+            </li>
+
+            <li>
+              Concurrent client requests. Each Impala daemon can handle multiple concurrent client requests. The
+              effects on performance depend on your particular hardware and workload.
+            </li>
+
+            <li>
+              Kerberos authentication. For more information, see
+              <xref href="impala_security.xml#security"/>.
+            </li>
+
+            <li>
+              Partitions. With Impala SQL, you can create partitioned tables with the <codeph>CREATE TABLE</codeph>
+              statement, and add and drop partitions with the <codeph>ALTER TABLE</codeph> statement. Impala also
+              takes advantage of the partitioning present in Hive tables. See
+              <xref href="impala_partitioning.xml#partitioning"/> for details.
+            </li>
+          </ul>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_unsupported">
+
+        <title>What features from relational databases or Hive are not available in Impala?</title>
+
+        <sectiondiv id="faq_unsupported_sql">
+
+          <!-- To do:
+            Good opportunity for a conref since there is a similar "unsupported" topic in the Language Reference section.
+          -->
+
+          <ul>
+            <li>
+              Querying streaming data.
+            </li>
+
+            <li>
+              Deleting individual rows. You delete data in bulk by overwriting an entire table or partition, or by
+              dropping a table.
+            </li>
+
+            <li>
+              Indexing (not currently). LZO-compressed text files can be indexed outside of Impala, as described in
+              <xref href="impala_txtfile.xml#lzo"/>.
+            </li>
+
+<!--
+          <li>
+            YARN integration (available when Impala is used with CDH 5).
+          </li>
+-->
+
+            <li>
+<!-- Former URL disappeared: cloudera.comcloudera/en/products/cdh/search.html -->
+<!-- Subscription URL doesn't seem appropriate: http://www.cloudera.com/content/cloudera/en/products-and-services/cloudera-enterprise/RTS-subscription.html -->
+              Full text search on text fields. The Cloudera Search product is appropriate for this use case.
+            </li>
+
+            <li>
+              Custom Hive Serializer/Deserializer classes (SerDes). Impala supports a set of common native file
+              formats that have built-in SerDes in CDH. See <xref href="impala_file_formats.xml#file_formats"/> for
+              details.
+            </li>
+
+            <li>
+              Checkpointing within a query. That is, Impala does not save intermediate results to disk during
+              long-running queries. Currently, Impala cancels a running query if any host on which that query is
+              executing fails. When one or more hosts are down, Impala reroutes future queries to only use the
+              available hosts, and Impala detects when the hosts come back up and begins using them again. Because
+              a query can be submitted through any Impala node, there is no single point of failure. In the future,
+              we will consider adding additional work allocation features to Impala, so that a running query would
+              complete even in the presence of host failures.
+            </li>
+
+<!--
+          <li>
+            Transforms.
+          </li>
+-->
+
+            <li>
+              Encryption of data transmitted between Impala daemons.
+            </li>
+
+<!--
+            <li>
+              Window functions.
+            </li>
+-->
+
+<!--
+          <li>
+            Hive UDFs.
+          </li>
+-->
+
+            <li>
+              Hive indexes.
+            </li>
+
+            <li>
+              Non-Hadoop data stores, such as relational databases.
+            </li>
+          </ul>
+
+          <p>
+            For the detailed list of features that are different between Impala and HiveQL, see
+            <xref href="impala_langref_unsupported.xml#langref_hiveql_delta"/>.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_jdbc">
+
+        <title>Does Impala support generic JDBC?</title>
+
+        <sectiondiv id="faq_jdbc_sect">
+
+          <p>
+            Impala supports the HiveServer2 JDBC driver.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_avro">
+
+        <title>Is Avro supported?</title>
+
+        <sectiondiv id="faq_avro_sect">
+
+          <p>
+            Yes, Avro is supported. Impala has always been able to query Avro tables. You can use the Impala
+            <codeph>LOAD DATA</codeph> statement to load existing Avro data files into a table. Starting with
+            Impala 1.4, you can create Avro tables with Impala. Currently, you still use the
+            <codeph>INSERT</codeph> statement in Hive to copy data from another table into an Avro table. See
+            <xref href="impala_avro.xml#avro"/> for details.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section audience="Cloudera" id="faq_roadmap">
+
+<!-- Hidden to avoid RevRec implications. -->
+
+        <title>What's next for Impala?</title>
+
+        <sectiondiv id="faq_next">
+
+          <p>
+            See our blog post:
+            <xref href="http://blog.cloudera.com/blog/2013/09/whats-next-for-impala-after-release-1-1/" scope="external" format="html">http://blog.cloudera.com/blog/2012/12/whats-next-for-cloudera-impala/</xref>
+          </p>
+
+        </sectiondiv>
+      </section>
+    </conbody>
+  </concept>
+
+  <concept id="faq_tasks">
+
+    <title>How do I?</title>
+
+    <conbody>
+
+      <p outputclass="toc inpage" audience="PDF">
+        FAQs in this category:
+      </p>
+
+      <section id="faq_secure_sql_text">
+
+        <title>How do I prevent users from seeing the text of SQL queries?</title>
+
+        <p>
+          For instructions on making the Impala log files unreadable by unprivileged users, see
+          <xref href="impala_security_files.xml#secure_files"/>.
+        </p>
+
+        <p>
+          For instructions on password-protecting the web interface to the Impala log files and other internal
+          server information, see <xref href="impala_security_webui.xml#security_webui"/>.
+        </p>
+
+        <p rev="2.2.0">
+          In <keyword keyref="impala22_full"/> and higher, you can use the log redaction feature
+          to obfuscate sensitive information in Impala log files.
+          See
+          <xref audience="integrated" href="sg_redaction.xml#log_redact"/><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/sg_redaction.html" scope="external" format="html"/>
+          for details.
+        </p>
+
+      </section>
+
+      <section id="faq_num_nodes">
+
+        <title>How do I know how many Impala nodes are in my cluster?</title>
+
+        <p>
+          The Impala statestore keeps track of how many <cmdname>impalad</cmdname> nodes are currently available.
+          You can see this information through the statestore web interface. For example, at the URL
+          <codeph>http://<varname>statestore_host</varname>:25010/metrics</codeph> you might see lines like the
+          following:
+        </p>
+
+<codeblock>statestore.live-backends:3
+statestore.live-backends.list:[<varname>host1</varname>:22000, <varname>host1</varname>:26000, <varname>host2</varname>:22000]</codeblock>
+
+        <p>
+          The number of <cmdname>impalad</cmdname> nodes is the number of list items referring to port 22000, in
+          this case two. (Typically, this number is one less than the number reported by the
+          <codeph>statestore.live-backends</codeph> line.) If an <cmdname>impalad</cmdname> node became unavailable
+          or came back after an outage, the information reported on this page would change appropriately.
+        </p>
+
+        <!-- To do:
+          If there is a good CM technique, mention that here also.
+        -->
+      </section>
+
+    </conbody>
+  </concept>
+
+  <concept id="faq_performance">
+
+    <title>Impala Performance</title>
+
+    <conbody>
+
+<!-- Template for new FAQ entries.
+      <section>
+        <title></title>
+        <sectiondiv id="">
+        <p>
+        </p>
+        </sectiondiv>
+      </section>
+
+-->
+
+      <p outputclass="toc inpage" audience="PDF">
+        FAQs in this category:
+      </p>
+
+      <section id="faq_streaming">
+
+        <title>Are results returned as they become available, or all at once when a query completes?</title>
+
+        <sectiondiv id="faq_stream_results">
+
+          <p>
+            Impala streams results whenever they are available, when possible. Certain SQL operations (aggregation
+            or <codeph>ORDER BY</codeph>) require all of the input to be ready before Impala can return results.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_slow_query">
+
+        <title>Why does my query run slowly?</title>
+
+        <sectiondiv id="faq_slow_query_sect">
+
+          <p>
+            There are many possible reasons why a given query could be slow. Use the following checklist to
+            diagnose performance issues with existing queries, and to avoid such issues when writing new queries,
+            setting up new nodes, creating new tables, or loading data.
+          </p>
+
+          <ul>
+            <li rev="1.4.0">
+              Immediately after the query finishes, issue a <codeph>SUMMARY</codeph> command in
+              <cmdname>impala-shell</cmdname>. You can check which phases of execution took the longest, and
+              compare estimated values for memory usage and number of rows with the actual values.
+            </li>
+
+            <li>
+              Immediately after the query finishes, issue a <codeph>PROFILE</codeph> command in
+              <cmdname>impala-shell</cmdname>. The numbers in the <codeph>BytesRead</codeph>,
+              <codeph>BytesReadLocal</codeph>, and <codeph>BytesReadShortCircuit</codeph> should be identical for a
+              specific node. For example:
+<codeblock>- BytesRead: 180.33 MB
+- BytesReadLocal: 180.33 MB
+- BytesReadShortCircuit: 180.33 MB</codeblock>
+              If <codeph>BytesReadLocal</codeph> is lower than <codeph>BytesRead</codeph>, something in your
+              cluster is misconfigured, such as the <cmdname>impalad</cmdname> daemon not running on all the data
+              nodes. If <codeph>BytesReadShortCircuit</codeph> is lower than <codeph>BytesRead</codeph>,
+              short-circuit reads are not enabled properly on that node; see
+              <xref href="impala_config_performance.xml#config_performance"/> for instructions.
+            </li>
+
+            <li>
+              If the table was just created, or this is the first query that accessed the table after an
+              <codeph>INVALIDATE METADATA</codeph> statement or after the <cmdname>impalad</cmdname> daemon was
+              restarted, there might be a one-time delay while the metadata for the table is loaded and cached.
+              Check whether the slowdown disappears when the query is run again. When doing performance
+              comparisons, consider issuing a <codeph>DESCRIBE <varname>table_name</varname></codeph> statement for
+              each table first, to make sure any timings only measure the actual query time and not the one-time
+              wait to load the table metadata.
+            </li>
+
+            <li>
+              Is the table data in uncompressed text format? Check by issuing a <codeph>DESCRIBE FORMATTED
+              <varname>table_name</varname></codeph> statement. A text table is indicated by the line:
+<codeblock>InputFormat: org.apache.hadoop.mapred.TextInputFormat</codeblock>
+              Although uncompressed text is the default format for a <codeph>CREATE TABLE</codeph> statement with
+              no <codeph>STORED AS</codeph> clauses, it is also the bulkiest format for disk storage and
+              consequently usually the slowest format for queries. For data where query performance is crucial,
+              particularly for tables that are frequently queried, consider starting with or converting to a
+              compact binary file format such as Parquet, Avro, RCFile, or SequenceFile. For details, see
+              <xref href="impala_file_formats.xml#file_formats"/>.
+            </li>
+
+            <li>
+              If your table has many columns, but the query refers to only a few columns, consider using the
+              Parquet file format. Its data files are organized with a column-oriented layout that lets queries
+              minimize the amount of I/O needed to retrieve, filter, and aggregate the values for specific columns.
+              See <xref href="impala_parquet.xml#parquet"/> for details.
+            </li>
+
+            <li>
+              If your query involves any joins, are the tables in the query ordered so that the tables or
+              subqueries are ordered with the one returning the largest number of rows on the left, followed by the
+              smallest (most selective), the second smallest, and so on? That ordering allows Impala to optimize
+              the way work is distributed among the nodes and how intermediate results are routed from one node to
+              another. For example, all other things being equal, the following join order results in an efficient
+              query:
+<codeblock>select some_col from
+    huge_table join big_table join small_table join medium_table
+  where
+    huge_table.id = big_table.id
+    and big_table.id = medium_table.id
+    and medium_table.id = small_table.id;</codeblock>
+              See <xref href="impala_perf_joins.xml#perf_joins"/> for performance tips for join queries.
+            </li>
+
+            <li>
+              Also for join queries, do you have table statistics for the table, and column statistics for the
+              columns used in the join clauses? Column statistics let Impala better choose how to distribute the
+              work for the various pieces of a join query. See <xref href="impala_perf_stats.xml#perf_stats"/> for
+              details about gathering statistics.
+            </li>
+
+            <li>
+              Does your table consist of many small data files? Impala works most efficiently with data files in
+              the multi-megabyte range; Parquet, a format optimized for data warehouse-style queries, uses
+              <ph rev="parquet_block_size">large files (originally 1 GB, now 256 MB in Impala 2.0 and higher) with
+              a block size matching the file size</ph>. Use the <codeph>DESCRIBE FORMATTED
+              <varname>table_name</varname></codeph> statement in <cmdname>impala-shell</cmdname> to see where the
+              data for a table is located, and use the <cmdname>hadoop fs -ls</cmdname> or <cmdname>hdfs dfs
+              -ls</cmdname> Unix commands to see the files and their sizes. If you have thousands of small data
+              files, that is a signal that you should consolidate into a smaller number of large files. Use an
+              <codeph>INSERT ... SELECT</codeph> statement to copy the data to a new table, reorganizing into new
+              data files as part of the process. Prefer to construct large data files and import them in bulk
+              through the <codeph>LOAD DATA</codeph> or <codeph>CREATE EXTERNAL TABLE</codeph> statements, rather
+              than issuing many <codeph>INSERT ... VALUES</codeph> statements; each <codeph>INSERT ...
+              VALUES</codeph> statement creates a separate tiny data file. If you have thousands of files all in
+              the same directory, but each one is megabytes in size, consider using a partitioned table so that
+              each partition contains a smaller number of files. See the following point for more on partitioning.
+            </li>
+
+            <li>
+              If your data is easy to group according to time or geographic region, have you partitioned your table
+              based on the corresponding columns such as <codeph>YEAR</codeph>, <codeph>MONTH</codeph>, and/or
+              <codeph>DAY</codeph>? Partitioning a table based on certain columns allows queries that filter based
+              on those same columns to avoid reading the data files for irrelevant years, postal codes, and so on.
+              (Do not partition down to too fine a level; try to structure the partitions so that there is still
+              sufficient data in each one to take advantage of the multi-megabyte HDFS block size.) See
+              <xref href="impala_partitioning.xml#partitioning"/> for details.
+            </li>
+          </ul>
+
+        </sectiondiv>
+      </section>
+
+      <section id="failed_query">
+
+        <title>Why does my SELECT statement fail?</title>
+
+        <sectiondiv id="faq_select_fail">
+
+          <p>
+            When a <codeph>SELECT</codeph> statement fails, the cause usually falls into one of the following
+            categories:
+          </p>
+
+          <ul>
+            <li>
+              A timeout because of a performance, capacity, or network issue affecting one particular node.
+            </li>
+
+            <li>
+              Excessive memory use for a join query, resulting in automatic cancellation of the query.
+            </li>
+
+            <li>
+              A low-level issue affecting how native code is generated on each node to handle particular
+              <codeph>WHERE</codeph> clauses in the query. For example, a machine instruction could be generated
+              that is not supported by the processor of a certain node. If the error message in the log suggests
+              the cause was an illegal instruction, consider turning off native code generation temporarily, and
+              trying the query again.
+            </li>
+
+            <li>
+              Malformed input data, such as a text data file with an enormously long line, or with a delimiter that
+              does not match the character specified in the <codeph>FIELDS TERMINATED BY</codeph> clause of the
+              <codeph>CREATE TABLE</codeph> statement.
+            </li>
+          </ul>
+
+        </sectiondiv>
+      </section>
+
+      <section id="failed_insert">
+
+        <title>Why does my INSERT statement fail?</title>
+
+        <sectiondiv id="faq_insert_fail">
+
+          <p>
+            When an <codeph>INSERT</codeph> statement fails, it is usually the result of exceeding some limit
+            within a Hadoop component, typically HDFS.
+          </p>
+
+          <ul>
+            <li>
+              An <codeph>INSERT</codeph> into a partitioned table can be a strenuous operation due to the
+              possibility of opening many files and associated threads simultaneously in HDFS. Impala 1.1.1
+              includes some improvements to distribute the work more efficiently, so that the values for each
+              partition are written by a single node, rather than as a separate data file from each node.
+            </li>
+
+            <li>
+              Certain expressions in the <codeph>SELECT</codeph> part of the <codeph>INSERT</codeph> statement can
+              complicate the execution planning and result in an inefficient <codeph>INSERT</codeph> operation. Try
+              to make the column data types of the source and destination tables match up, for example by doing
+              <codeph>ALTER TABLE ... REPLACE COLUMNS</codeph> on the source table if necessary. Try to avoid
+              <codeph>CASE</codeph> expressions in the <codeph>SELECT</codeph> portion, because they make the
+              result values harder to predict than transferring a column unchanged or passing the column through a
+              built-in function.
+            </li>
+
+            <li>
+              Be prepared to raise some limits in the HDFS configuration settings, either temporarily during the
+              <codeph>INSERT</codeph> or permanently if you frequently run such <codeph>INSERT</codeph> statements
+              as part of your ETL pipeline.
+            </li>
+
+            <li>
+              The resource usage of an <codeph>INSERT</codeph> statement can vary depending on the file format of
+              the destination table. Inserting into a Parquet table is memory-intensive, because the data for each
+              partition is buffered in memory until it reaches 1 gigabyte, at which point the data file is written
+              to disk. Impala can distribute the work for an <codeph>INSERT</codeph> more efficiently when
+              statistics are available for the source table that is queried during the <codeph>INSERT</codeph>
+              statement. See <xref href="impala_perf_stats.xml#perf_stats"/> for details about gathering
+              statistics.
+            </li>
+          </ul>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_scalability">
+
+        <title>Does Impala performance improve as it is deployed to more hosts in a cluster in much the same way that Hadoop performance does?</title>
+
+        <sectiondiv id="faq_hosts">
+
+          <draft-comment translate="no">
+Like to combine this one with the DataNodes question a little later.
+</draft-comment>
+
+          <p>
+            Yes. Impala scales with the number of hosts. It is important to install Impala on all the DataNodes in
+            the cluster, because otherwise some of the nodes must do remote reads to retrieve data not available
+            for local reads. Data locality is an important architectural aspect for Impala performance. See
+            <xref href="http://blog.cloudera.com/blog/2014/01/impala-performance-dbms-class-speed/" scope="external" format="html">this
+            Impala performance blog post</xref> for background. Note that this blog post refers to benchmarks with
+            Impala 1.1.1; Impala has added even more performance features in the 1.2.x series.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_hdfs_block_size">
+
+        <title>Is the HDFS block size reduced to achieve faster query results?</title>
+
+        <sectiondiv id="faq_block_size">
+
+          <p>
+            No. Impala does not make any changes to the HDFS or HBase data sets.
+          </p>
+
+          <p>
+            The default Parquet block size is relatively large (<ph rev="parquet_block_size">256 MB in Impala 2.0
+            and later; 1 GB in earlier releases</ph>). You can control the block size when creating Parquet files
+            using the <xref href="impala_parquet_file_size.xml#parquet_file_size">PARQUET_FILE_SIZE</xref> query
+            option.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_caching">
+
+        <title>Does Impala use caching?</title>
+
+        <sectiondiv>
+
+          <p id="caching">
+            Impala does not cache table data. It does cache some table and file metadata. Although queries might run
+            faster on subsequent iterations because the data set was cached in the OS buffer cache, Impala does not
+            explicitly control this.
+          </p>
+
+          <p rev="1.4.0">
+            Impala takes advantage of the HDFS caching feature in CDH 5. You can designate
+            which tables or partitions are cached through the <codeph>CACHED</codeph>
+            and <codeph>UNCACHED</codeph> clauses of the <codeph>CREATE TABLE</codeph>
+            and <codeph>ALTER TABLE</codeph> statements.
+            Impala can also take advantage of data that is pinned in the HDFS cache
+            through the <cmdname>hdfscacheadmin</cmdname> command.
+            See <xref href="impala_perf_hdfs_caching.xml#hdfs_caching"/> for details.
+          </p>
+
+        </sectiondiv>
+      </section>
+    </conbody>
+  </concept>
+
+  <concept id="faq_use_cases">
+
+    <title>Impala Use Cases</title>
+    <prolog>
+      <metadata>
+        <data name="Category" value="Use Cases"/>
+      </metadata>
+    </prolog>
+
+    <conbody>
+
+      <p outputclass="toc inpage" audience="PDF">
+        FAQs in this category:
+      </p>
+
+      <section id="faq_impala_hive_mr">
+
+        <title>What are good use cases for Impala as opposed to Hive or MapReduce?</title>
+
+        <sectiondiv id="faq_impala_vs_hive">
+
+          <p>
+            Impala is well-suited to executing SQL queries for interactive exploratory analytics on large data
+            sets. Hive and MapReduce are appropriate for very long running, batch-oriented tasks such as ETL.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_mapreduce">
+
+        <title>Is MapReduce required for Impala? Will Impala continue to work as expected if MapReduce is stopped?</title>
+
+        <sectiondiv id="faq_mapreduce_sect">
+
+          <p>
+            Impala does not use MapReduce at all.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_cep">
+
+        <title>Can Impala be used for complex event processing?</title>
+
+        <sectiondiv id="faq_cep_sect">
+
+          <p>
+            For example, in an industrial environment, many agents may generate large amounts of data. Can Impala
+            be used to analyze this data, checking for notable changes in the environment?
+          </p>
+
+          <p>
+            Complex Event Processing (CEP) is usually performed by dedicated stream-processing systems. Impala is
+            not a stream-processing system, as it most closely resembles a relational database.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_ad_hoc">
+
+        <title>Is Impala intended to handle real time queries in low-latency applications or is it for ad hoc queries for the purpose of data exploration?</title>
+
+        <sectiondiv id="faq_real_time">
+
+          <p>
+            Ad-hoc queries are the primary use case for Impala. We anticipate it being used in many other
+            situations where low-latency is required. Whether Impala is appropriate for any particular use-case
+            depends on the workload, data size and query volume. See <xref href="impala_intro.xml#benefits"/> for
+            the primary benefits you can expect when using Impala.
+          </p>
+
+        </sectiondiv>
+      </section>
+    </conbody>
+  </concept>
+
+  <concept id="faq_hive">
+
+    <title>Questions about Impala And Hive</title>
+
+    <conbody>
+
+      <p outputclass="toc inpage" audience="PDF">
+        FAQs in this category:
+      </p>
+
+      <draft-comment translate="no">
+Note: earlier question refers to Impala vs. Hive and MapReduce altogether.
+Should consolidate since makes sense to have one faq_hive ID.
+</draft-comment>
+
+      <section id="faq_hive_pig">
+
+        <title>How does Impala compare to Hive and Pig?</title>
+
+        <sectiondiv id="faq_hive_pig_sect">
+
+          <p>
+            Impala is different from Hive and Pig because it uses its own daemons that are spread across the
+            cluster for queries. Because Impala does not rely on MapReduce, it avoids the startup overhead of
+            MapReduce jobs, allowing Impala to return results in real time.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_serdes">
+
+        <title>Can I do transforms or add new functionality?</title>
+
+        <sectiondiv id="faq_udf">
+
+          <p>
+            Impala adds support for UDFs in Impala 1.2. You can write your own functions in C++, or reuse existing
+            Java-based Hive UDFs. The UDF support includes scalar functions and user-defined aggregate functions
+            (UDAs). User-defined table functions (UDTFs) are not currently supported.
+          </p>
+
+          <p>
+            Impala does not currently support an extensible serialization-deserialization framework (SerDes), and
+            so adding extra functionality to Impala is not as straightforward as for Hive or Pig.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_hive_compat">
+
+        <title>Can any Impala query also be executed in Hive?</title>
+
+        <sectiondiv id="faq_hiveql">
+
+          <p>
+            Yes. There are some minor differences in how some queries are handled, but Impala queries can also be
+            completed in Hive. Impala SQL is a subset of HiveQL, with some functional limitations such as
+            transforms. For details of the Impala SQL dialect, see
+            <xref href="impala_langref_sql.xml#langref_sql"/>. For the Impala built-in functions, see
+            <xref href="impala_functions.xml#builtins"/>. For the detailed list of differences between Impala and
+            HiveQL, see <xref href="impala_langref_unsupported.xml#langref_hiveql_delta"/>.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_hive_hbase_import">
+
+        <title>Can I use Impala to query data already loaded into Hive and HBase?</title>
+
+        <sectiondiv id="faq_hive_hbase">
+
+          <p>
+            There are no additional steps to allow Impala to query tables managed by Hive, whether they are stored
+            in HDFS or HBase. Make sure that Impala is configured to access the Hive metastore correctly and you
+            should be ready to go. Keep in mind that <codeph>impalad</codeph>, by default, runs as the
+            <codeph>impala</codeph> user, so you might need to adjust some file permissions depending on how strict
+            your permissions are currently.
+          </p>
+
+          <p>
+            See <xref href="impala_hbase.xml#impala_hbase"/> for details about querying data in HBase.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_hive_prereq">
+
+        <title>Is Hive an Impala requirement?</title>
+
+        <sectiondiv id="faq_hive_prereq_sect">
+
+          <p>
+            The Hive metastore service is a requirement. Impala shares the same metastore database as Hive,
+            allowing Impala and Hive to access the same tables transparently.
+          </p>
+
+          <p>
+            Hive itself is optional, and does not need to be installed on the same nodes as Impala. Currently,
+            Impala supports a wider variety of read (query) operations than write (insert) operations; you use Hive
+            to insert data into tables that use certain file formats. See
+            <xref href="impala_file_formats.xml#file_formats"/> for details.
+          </p>
+
+        </sectiondiv>
+      </section>
+    </conbody>
+  </concept>
+
+  <concept id="faq_ha">
+
+    <title>Impala Availability</title>
+
+    <conbody>
+
+      <p outputclass="toc inpage" audience="PDF">
+        FAQs in this category:
+      </p>
+
+      <section id="faq_production">
+
+        <title>Is Impala production ready?</title>
+
+        <sectiondiv id="faq_production_sect">
+
+          <p>
+            Impala has finished its beta release cycle, and the 1.0, 1.1, and 1.2 GA releases are production ready.
+            The 1.1.x series includes additional security features for authorization, an important requirement for
+            production use in many organizations. The 1.2.x series includes important performance features,
+            particularly for large join queries. Some Cloudera customers are already using Impala for large
+            workloads.
+          </p>
+
+          <p rev="1.3.0">
+            The Impala 1.3.0 and higher releases are bundled with corresponding levels of CDH 5.
+            The number of new features grows with each release.
+            See <xref href="impala_new_features.xml#new_features"/> for a full list.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_ha_config">
+
+        <title>How do I configure Hadoop high availability (HA) for Impala?</title>
+
+        <sectiondiv id="faq_ha_config_sect">
+
+          <p rev="1.2.0">
+            You can set up a proxy server to relay requests back and forth to the Impala servers, for load
+            balancing and high availability. See <xref href="impala_proxy.xml#proxy"/> for details.
+          </p>
+
+          <p>
+            You can enable HDFS HA for the Hive metastore. See the
+            <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_hag_cdh_other_ha.html" scope="external" format="html">CDH5 High Availability Guide</xref>
+            for details.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_spof">
+
+        <title>What happens if there is an error in Impala?</title>
+
+        <sectiondiv id="faq_spof_sect">
+
+          <p>
+            There is not a single point of failure in Impala. All Impala daemons are fully able to handle incoming
+            queries. If a machine fails however, all queries with fragments running on that machine will fail.
+            Because queries are expected to return quickly, you can just rerun the query if there is a failure. See
+            <xref href="impala_concepts.xml#concepts"/> for details about the Impala architecture.
+          </p>
+
+          <draft-comment translate="no">
+Clarify to what extent the catalog service could be seen as a single point of failure.
+</draft-comment>
+
+          <p>
+            The longer answer: Impala must be able to connect to the Hive metastore. Impala aggressively caches
+            metadata so the metastore host should have minimal load. Impala relies on the HDFS NameNode, and, in
+            CDH4, you can configure HA for HDFS. Impala also has centralized services, known as the
+            <xref href="impala_components.xml#intro_statestore">statestore</xref> and
+            <xref href="impala_components.xml#intro_catalogd">catalog</xref> services, that run on one host only.
+            Impala continues to execute queries if the statestore host is down, but it will not get state updates.
+            For example, if a host is added to the cluster while the statestore host is down, the existing
+            instances of <codeph>impalad</codeph> running on the other hosts will not find out about this new host.
+            Once the statestore process is restarted, all the information it serves is automatically reconstructed
+            from all running Impala daemons.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_max_rows">
+
+        <title>What is the maximum number of rows in a table?</title>
+
+        <sectiondiv id="faq_max_rows_sect">
+
+          <p>
+            There is no defined maximum. Some customers have used Impala to query a table with over a trillion
+            rows.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_contention">
+
+        <title>Can Impala and MapReduce jobs run on the same cluster without resource contention?</title>
+
+        <sectiondiv id="faq_mapreduce_contention">
+
+          <p>
+            Yes. See <xref href="impala_perf_resources.xml#mem_limits"/> for how to control Impala resource usage
+            using the Linux cgroup mechanism, and <xref href="impala_resource_management.xml#resource_management"/>
+            for how to use Impala with the YARN resource management framework. Impala is designed to run on the
+            DataNode hosts. Any contention depends mostly on the cluster setup and workload.
+          </p>
+
+          <p conref="../shared/impala_common.xml#common/impala_mr"/>
+
+        </sectiondiv>
+      </section>
+    </conbody>
+  </concept>
+
+  <concept id="faq_internals">
+
+    <title>Impala Internals</title>
+
+    <conbody>
+
+      <p outputclass="toc inpage" audience="PDF">
+        FAQs in this category:
+      </p>
+
+      <section id="faq_impalad_hosts">
+
+        <title>On which hosts does Impala run?</title>
+
+        <sectiondiv id="faq_data_nodes">
+
+          <p>
+            Cloudera strongly recommends running the <cmdname>impalad</cmdname> daemon on each DataNode for good
+            performance. Although this topology is not a hard requirement, if there are data blocks with no Impala
+            daemons running on any of the hosts containing replicas of those blocks, queries involving that data
+            could be very inefficient. In that case, the data must be transmitted from one host to another for
+            processing by <q>remote reads</q>, a condition Impala normally tries to avoid. See
+            <xref href="impala_concepts.xml#concepts"/> for details about the Impala architecture. Impala schedules
+            query fragments on all hosts holding data relevant to the query, if possible.
+          </p>
+
+          <p>
+            In cases where some hosts in the cluster have much greater CPU and memory capacity than others, or
+            where some hosts have extra CPU capacity because some CPU-intensive phases are single-threaded,
+            some users have run multiple <cmdname>impalad</cmdname> daemons on a single host to take advantage
+            of the extra CPU capacity. This configuration is only practical for specific workloads that
+            rely heavily on aggregation, and the physical hosts must have sufficient memory to accomodate
+            the requirements for multiple <cmdname>impalad</cmdname> instances.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_join_internals">
+
+        <title>How are joins performed in Impala?</title>
+
+        <sectiondiv id="faq_joins">
+
+          <draft-comment translate="no">
+Will change with join order optimizations, now slated for 1.2.2.
+</draft-comment>
+
+          <p>
+            By default, Impala automatically determines the most efficient order in which to join tables using a
+            cost-based method, based on their overall size and number of rows. (This is a new feature in Impala
+            1.2.2 and higher.) The <codeph>COMPUTE STATS</codeph> statement gathers information about each table
+            that is crucial for efficient join performance.
+<!--
+          The order in which tables are joined is the same order in which tables appear in the
+          <codeph>SELECT</codeph> statement's
+          <codeph>FROM</codeph> clause. That is, there is no join order optimization
+          taking place at the moment. It is usually optimal for the smallest table to appear as the right-most table in
+          a <codeph>JOIN</codeph> clause.
+          -->
+            Impala chooses between two techniques for join queries, known as <q>broadcast joins</q> and
+            <q>partitioned joins</q>. See <xref href="impala_joins.xml#joins"/> for syntax details and
+            <xref href="impala_perf_joins.xml#perf_joins"/> for performance considerations.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_join_sizes">
+
+        <title>How does Impala process join queries for large tables?</title>
+
+        <sectiondiv>
+
+          <p>
+            Impala utilizes multiple strategies to allow joins between tables and result sets of various sizes.
+            When joining a large table with a small one, the data from the small table is transmitted to each node
+            for intermediate processing. When joining two large tables, the data from one of the tables is divided
+            into pieces, and each node processes only selected pieces. See <xref href="impala_joins.xml#joins"/>
+            for details about join processing, <xref href="impala_perf_joins.xml#perf_joins"/> for performance
+            considerations, and <xref href="impala_hints.xml#hints"/> for how to fine-tune the join strategy.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_aggregation_implementation">
+
+        <title>What is Impala's aggregation strategy?</title>
+
+        <sectiondiv id="faq_join_aggregation">
+
+          <p rev="2.0.0">
+            Impala currently only supports in-memory hash aggregation.
+            In Impala 2.0 and higher, if the memory requirements for a
+            join or aggregation operation exceed the memory limit for
+            a particular host, Impala uses a temporary work area on disk
+            to help the query complete successfully.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_metadata_management">
+
+        <title>How is Impala metadata managed?</title>
+
+        <sectiondiv id="faq_metadata">
+
+          <draft-comment translate="no">
+Doesn't seem related to joins...
+</draft-comment>
+
+          <p>
+            Impala uses two pieces of metadata: the catalog information from the Hive metastore and the file
+            metadata from the NameNode. Currently, this metadata is lazily populated and cached when an
+            <codeph>impalad</codeph> needs it to plan a query.
+          </p>
+
+          <p>
+            The <xref href="impala_refresh.xml#refresh">REFRESH</xref> statement updates the metadata for a
+            particular table after loading new data through Hive. The
+            <xref href="impala_invalidate_metadata.xml#invalidate_metadata"/> statement refreshes all metadata, so
+            that Impala recognizes new tables or other DDL and DML changes performed through Hive.
+          </p>
+
+          <p rev="1.2.0">
+            In Impala 1.2 and higher, a dedicated <cmdname>catalogd</cmdname> daemon broadcasts metadata changes
+            due to Impala DDL or DML statements to all nodes, reducing or eliminating the need to use the
+            <codeph>REFRESH</codeph> and <codeph>INVALIDATE METADATA</codeph> statements.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_namenode_overhead">
+
+        <title>What load do concurrent queries produce on the NameNode?</title>
+
+        <sectiondiv id="faq_namenode_load">
+
+          <p>
+            The load Impala generates is very similar to MapReduce. Impala contacts the NameNode during the
+            planning phase to get the file metadata (this is only run on the host the query was sent to). Every
+            <codeph>impalad</codeph> will read files as part of normal processing of the query.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_perf_architecture">
+
+        <title>How does Impala achieve its performance improvements?</title>
+
+        <sectiondiv id="faq_performance_features">
+
+          <p>
+            These are the main factors in the performance of Impala versus that of other Hadoop components and
+            related technologies.
+          </p>
+
+          <p>
+            Impala avoids MapReduce. While MapReduce is a great general parallel processing model with many
+            benefits, it is not designed to execute SQL. Impala avoids the inefficiencies of MapReduce in these
+            ways:
+          </p>
+
+          <ul>
+            <li>
+              Impala does not materialize intermediate results to disk. SQL queries often map to multiple MapReduce
+              jobs with all intermediate data sets written to disk.
+            </li>
+
+            <li>
+              Impala avoids MapReduce start-up time. For interactive queries, the MapReduce start-up time becomes
+              very noticeable. Impala runs as a service and essentially has no start-up time.
+            </li>
+
+            <li>
+              Impala can more naturally disperse query plans instead of having to fit them into a pipeline of map
+              and reduce jobs. This enables Impala to parallelize multiple stages of a query and avoid overheads
+              such as sort and shuffle when unnecessary.
+            </li>
+          </ul>
+
+          <p>
+            Impala uses a more efficient execution engine by taking advantage of modern hardware and technologies:
+          </p>
+
+          <ul>
+            <li>
+              Impala generates runtime code. Impala uses LLVM to generate assembly code for the query that is being
+              run. Individual queries do not have to pay the overhead of running on a system that needs to be able
+              to execute arbitrary queries.
+            </li>
+
+            <li>
+              Impala uses available hardware instructions when possible. Impala uses the supplemental SSE3 (SSSE3)
+              instructions which can offer tremendous speedups in some cases. (Impala 2.0 and 2.1 required
+              the SSE4.1 instruction set; Impala 2.2 and higher relax the restriction again so only
+              SSSE3 is required.)
+            </li>
+
+            <li>
+              Impala uses better I/O scheduling. Impala is aware of the disk location of blocks and is able to
+              schedule the order to process blocks to keep all disks busy.
+            </li>
+
+            <li>
+              Impala is designed for performance. A lot of time has been spent in designing Impala with sound
+              performance-oriented fundamentals, such as tight inner loops, inlined function calls, minimal
+              branching, better use of cache, and minimal memory usage.
+            </li>
+          </ul>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_memory_exceeded">
+
+        <title>What happens when the data set exceeds available memory?</title>
+
+        <sectiondiv id="faq_mem_limit_exceeded">
+
+          <p>
+            Currently, if the memory required to process intermediate results on a node exceeds the amount
+            available to Impala on that node, the query is cancelled. You can adjust the memory available to Impala
+            on each node, and you can fine-tune the join strategy to reduce the memory required for the biggest
+            queries. We do plan on supporting external joins and sorting in the future.
+          </p>
+
+          <p>
+            Keep in mind though that the memory usage is not directly based on the input data set size. For
+            aggregations, the memory usage is the number of rows <i>after</i> grouping. For joins, the memory usage
+            is the combined size of the tables <i>excluding</i> the biggest table, and Impala can use join
+            strategies that divide up large joined tables among the various nodes rather than transmitting the
+            entire table to each node.
+          </p>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_memory_pressure">
+
+        <title>What are the most memory-intensive operations?</title>
+
+        <sectiondiv id="faq_memory_fail">
+
+          <p>
+            If a query fails with an error indicating <q>memory limit exceeded</q>, you might suspect a memory
+            leak. The problem could actually be a query that is structured in a way that causes Impala to allocate
+            more memory than you expect, exceeded the memory allocated for Impala on a particular node. Some
+            examples of query or table structures that are especially memory-intensive are:
+          </p>
+
+          <ul>
+            <li>
+              <codeph>INSERT</codeph> statements using dynamic partitioning, into a table with many different
+              partitions. (Particularly for tables using Parquet format, where the data for each partition is held
+              in memory until it reaches <ph rev="parquet_block_size">the full block size</ph> in size before it is
+              written to disk.) Consider breaking up such operations into several different <codeph>INSERT</codeph>
+              statements, for example to load data one year at a time rather than for all years at once.
+            </li>
+
+            <li>
+              <codeph>GROUP BY</codeph> on a unique or high-cardinality column. Impala allocates some handler
+              structures for each different value in a <codeph>GROUP BY</codeph> query. Having millions of
+              different <codeph>GROUP BY</codeph> values could exceed the memory limit.
+            </li>
+
+            <li>
+              Queries involving very wide tables, with thousands of columns, particularly with many
+              <codeph>STRING</codeph> columns. Because Impala allows a <codeph>STRING</codeph> value to be up to 32
+              KB, the intermediate results during such queries could require substantial memory allocation.
+            </li>
+          </ul>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_memory_dealloc">
+
+        <title>When does Impala hold on to or return memory?</title>
+
+        <p>
+          Impala allocates memory using
+          <codeph><xref href="http://goog-perftools.sourceforge.net/doc/tcmalloc.html" scope="external" format="html">tcmalloc</xref></codeph>,
+          a memory allocator that is optimized for high concurrency. Once Impala allocates memory, it keeps that
+          memory reserved to use for future queries. Thus, it is normal for Impala to show high memory usage when
+          idle. If Impala detects that it is about to exceed its memory limit (defined by the
+          <codeph>-mem_limit</codeph> startup option or the <codeph>MEM_LIMIT</codeph> query option), it
+          deallocates memory not needed by the current queries.
+        </p>
+
+        <p>
+          When issuing queries through the JDBC or ODBC interfaces, make sure to call the appropriate close method
+          afterwards. Otherwise, some memory associated with the query is not freed.
+        </p>
+      </section>
+    </conbody>
+  </concept>
+
+  <concept id="faq_sql">
+
+    <title>SQL</title>
+
+    <conbody>
+
+      <p outputclass="toc inpage" audience="PDF">
+        FAQs in this category:
+      </p>
+
+      <section id="faq_update">
+
+        <title>Is there an UPDATE statement?</title>
+
+        <sectiondiv id="faq_update_sect">
+
+          <p>
+            Impala does not currently have an <codeph>UPDATE</codeph> statement, which would typically be used to
+            change a single row, a small group of rows, or a specific column. The HDFS-based files used by typical
+            Impala queries are optimized for bulk operations across many megabytes of data at a time, making
+            traditional <codeph>UPDATE</codeph> operations inefficient or impractical.
+          </p>
+
+          <p>
+            You can use the following techniques to achieve the same goals as the familiar <codeph>UPDATE</codeph>
+            statement, in a way that preserves efficient file layouts for subsequent queries:
+          </p>
+
+          <ul>
+            <li>
+              Replace the entire contents of a table or partition with updated data that you have already staged in
+              a different location, either using <codeph>INSERT OVERWRITE</codeph>, <codeph>LOAD DATA</codeph>, or
+              manual HDFS file operations followed by a <codeph>REFRESH</codeph> statement for the table.
+              Optionally, you can use built-in functions and expressions in the <codeph>INSERT</codeph> statement
+              to transform the copied data in the same way you would normally do in an <codeph>UPDATE</codeph>
+              statement, for example to turn a mixed-case string into all uppercase or all lowercase.
+            </li>
+
+            <li>
+              To update a single row, use an HBase table, and issue an <codeph>INSERT ... VALUES</codeph> statement
+              using the same key as the original row. Because HBase handles duplicate keys by only returning the
+              latest row with a particular key value, the newly inserted row effectively hides the previous one.
+            </li>
+          </ul>
+
+        </sectiondiv>
+      </section>
+
+      <section id="faq_udfs">
+
+        <title>Can Impala do user-defined functions (UDFs)?</title>
+
+        <p>
+          Impala 1.2 and higher does support UDFs and UDAs. You can either write native Impala UDFs and UDAs in
+          C++, or reuse UDFs (but not UDAs) originally written in Java for use with Hive. See
+          <xref href="impala_udf.xml#udfs"/> for details.
+        </p>
+      </section>
+
+      <section id="faq_refresh">
+
+        <title>Why do I have to use REFRESH and INVALIDATE METADATA, what do they do?</title>
+
+        <p>
+          In Impala 1.2 and higher, there is much less need to use the <codeph>REFRESH</codeph> and
+          <codeph>INVALIDATE METADATA</codeph> statements:
+        </p>
+
+        <ul>
+          <li>
+            The new <codeph>impala-catalog</codeph> service, represented by the <cmdname>catalogd</cmdname> daemon,
+            broadcasts the results of Impala DDL statements to all Impala nodes. Thus, if you do a <codeph>CREATE
+            TABLE</codeph> statement in Impala while connected to one node, you do not need to do
+            <codeph>INVALIDATE METADATA</codeph> before issuing queries through a different node.
+          </li>
+
+          <li>
+            The catalog service only recognizes changes made through Impala, so you must still issue a
+            <codeph>REFRESH</codeph> statement if you load data through Hive or by manipulating files in HDFS, and
+            you must issue an <codeph>INVALIDATE METADATA</codeph> statement if you create a table, alter a table,
+            add or drop partitions, or do other DDL statements in Hive.
+          </li>
+
+          <li>
+            Because the catalog service broadcasts the results of <codeph>REFRESH</codeph> and <codeph>INVALIDATE
+            METADATA</codeph> statements to all nodes, in the cases where you do still need to issue those
+            statements, you can do that on a single node rather than on every node, and the changes will be
+            automatically recognized across the cluster, making it more convenient to load balance by issuing
+            queries through arbitrary Impala nodes rather than always using the same coordinator node.
+          </li>
+        </ul>
+      </section>
+
+      <section id="faq_drop_table_space">
+
+        <title>Why is space not freed up when I issue DROP TABLE?</title>
+
+        <p>
+          Impala deletes data files when you issue a <codeph>DROP TABLE</codeph> on an internal table, but not an
+          external one. By default, the <codeph>CREATE TABLE</codeph> statement creates internal tables, where the
+          files are managed by Impala. An external table is created with a <codeph>CREATE EXTERNAL TABLE</codeph>
+          statement, where the files reside in a location outside the control of Impala. Issue a <codeph>DESCRIBE
+          FORMATTED</codeph> statement to check whether a table is internal or external. The keyword
+          <codeph>MANAGED_TABLE</codeph> indicates an internal table, from which Impala can delete the data files.
+          The keyword <codeph>EXTERNAL_TABLE</codeph> indicates an external table, where Impala will leave the data
+          files untouched when you drop the table.
+        </p>
+
+        <p>
+          Even when you drop an internal table and the files are removed from their original location, you might
+          not get the hard drive space back immediately. By default, files that are deleted in HDFS go into a
+          special trashcan directory, from which they are purged after a period of time (by default, 6 hours). For
+          background information on the trashcan mechanism, see
+          <xref href="https://archive.cloudera.com/cdh4/cdh/4/hadoop/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html" scope="external" format="html"/>.
+          For information on purging files from the trashcan, see
+          <xref href="https://archive.cloudera.com/cdh4/cdh/4/hadoop/hadoop-project-dist/hadoop-common/FileSystemShell.html" scope="external" format="html"/>.
+        </p>
+
+        <p>
+          When Impala deletes files and they are moved to the HDFS trashcan, they go into an HDFS directory owned
+          by the <codeph>impala</codeph> user. If the <codeph>impala</codeph> user does not have an HDFS home
+          directory where a trashcan can be created, the files are not deleted or moved, as a safety measure. If
+          you issue a <codeph>DROP TABLE</codeph> statement and find that the table data files are left in their
+          original location, create an HDFS directory <filepath>/user/impala</filepath>, owned and writeable by
+          the <codeph>impala</codeph> user. For example, you might find that <filepath>/user/impala</filepath> is
+          owned by the <codeph>hdfs</codeph> user, in which case you would switch to the <codeph>hdfs</codeph> user
+          and issue a command such as:
+        </p>
+
+<codeblock>hdfs dfs -chown -R impala /user/impala</codeblock>
+      </section>
+
+      <section id="faq_dual">
+
+        <title>Is there a DUAL table?</title>
+
+        <p>
+          You might be used to running queries against a single-row table named <codeph>DUAL</codeph> to try out
+          expressions, built-in functions, and UDFs. Impala does not have a <codeph>DUAL</codeph> table. To achieve
+          the same result, you can issue a <codeph>SELECT</codeph> statement without any table name:
+        </p>
+
+<codeblock>select 2+2;
+select substr('hello',2,1);
+select pow(10,6);
+</codeblock>
+      </section>
+    </conbody>
+  </concept>
+
+  <concept id="faq_partitioning">
+
+    <title>Partitioned Tables</title>
+
+    <conbody>
+
+      <p outputclass="toc inpage" audience="PDF">
+        FAQs in this category:
+      </p>
+
+      <section id="faq_partition_csv_etl">
+
+        <title>How do I load a big CSV file into a partitioned table?</title>
+
+        <p>
+          To load a data file into a partitioned table, when the data file includes fields like year, month, and so
+          on that correspond to the partition key columns, use a two-stage process. First, use the <codeph>LOAD
+          DATA</codeph> or <codeph>CREATE EXTERNAL TABLE</codeph> statement to bring the data into an unpartitioned
+          text table. Then use an <codeph>INSERT ... SELECT</codeph> statement to copy the data from the
+          unpartitioned table to a partitioned one. Include a <codeph>PARTITION</codeph> clause in the
+          <codeph>INSERT</codeph> statement to specify the partition key columns. The <codeph>INSERT</codeph>
+          operation splits up the data into separate data files for each partition. For examples, see
+          <xref href="impala_partitioning.xml#partitioning"/>. For details about loading data into partitioned
+          Parquet tables, a popular choice for high-volume data, see <xref href="impala_parquet.xml#parquet_etl"/>.
+        </p>
+      </section>
+
+      <section id="faq_partition_select_star">
+
+        <title>Can I do INSERT ... SELECT * into a partitioned table?</title>
+
+        <p>
+          When you use the <codeph>INSERT ... SELECT *</codeph> syntax to copy data into a partitioned table, the
+          columns corresponding to the partition key columns must appear last in the columns returned by the
+          <codeph>SELECT *</codeph>. You can create the table with the partition key columns defined last. Or, you
+          can use the <codeph>CREATE VIEW</codeph> statement to create a view that reorders the columns: put the
+          partition key columns last, then do the <codeph>INSERT ... SELECT *</codeph> from the view.
+        </p>
+      </section>
+    </conbody>
+  </concept>
+
+  <concept id="faq_hbase">
+
+    <title>HBase</title>
+
+    <conbody>
+
+      <p outputclass="toc inpage" audience="PDF">
+        FAQs in this category:
+      </p>
+
+      <section id="faq_hbase_use_cases">
+
+        <title>What kinds of Impala queries or data are best suited for HBase?</title>
+
+        <p>
+          HBase tables are ideal for queries where normally you would use a key-value store. That is, where you
+          retrieve a single row or a few rows, by testing a special unique key column using the <codeph>=</codeph>
+          or <codeph>IN</codeph> operators.
+        </p>
+
+        <p>
+          HBase tables are not suitable for queries that produce large result sets with thousands of rows. HBase
+          tables are also not suitable for queries that perform full table scans because the <codeph>WHERE</codeph>
+          clause does not request specific values from the unique key column.
+        </p>
+
+        <p>
+          Use HBase tables for data that is inserted one row or a few rows at a time, such as by the <codeph>INSERT
+          ... VALUES</codeph> syntax. Loading data piecemeal like this into an HDFS-backed table produces many tiny
+          files, which is a very inefficient layout for HDFS data files.
+        </p>
+
+        <p>
+          If the lack of an <codeph>UPDATE</codeph> statement in Impala is a problem for you, you can simulate
+          single-row updates by doing an <codeph>INSERT ... VALUES</codeph> statement using an existing value for
+          the key column. The old row value is hidden; only the new row value is seen by queries.
+        </p>
+
+        <p>
+          HBase tables are often wide (containing many columns) and sparse (with most column values
+          <codeph>NULL</codeph>). For example, you might record hundreds of different data points for each user of
+          an online service, such as whether the user had registered for an online game or enabled particular
+          account features. With Impala and HBase, you could look up all the information for a specific customer
+          efficiently in a single query. For any given customer, most of these columns might be
+          <codeph>NULL</codeph>, because a typical customer might not make use of most features of an online
+          service.
+        </p>
+      </section>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_faq_base.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_faq_base.xml b/docs/topics/impala_faq_base.xml
new file mode 100644
index 0000000..c999a1f
--- /dev/null
+++ b/docs/topics/impala_faq_base.xml
@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="obsolete_faq">
+
+  <title>Impala Frequently Asked Questions</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="FAQs"/>
+      <data name="Category" value="Planning"/>
+      <data name="Category" value="Getting Started"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <draft-comment translate="no">
+      Obsolete. Content all moved into impala_faq.xml.
+    </draft-comment>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_features.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_features.xml b/docs/topics/impala_features.xml
new file mode 100644
index 0000000..d219d15
--- /dev/null
+++ b/docs/topics/impala_features.xml
@@ -0,0 +1,21 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="features">
+
+  <title>Primary Impala Features</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Concepts"/>
+      <data name="Category" value="Getting Started"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p conref="../shared/impala_common.xml#common/feature_list"/>
+  </conbody>
+</concept>


[20/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_logging.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_logging.xml b/docs/topics/impala_logging.xml
new file mode 100644
index 0000000..960f1e6
--- /dev/null
+++ b/docs/topics/impala_logging.xml
@@ -0,0 +1,471 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="logging">
+
+  <title>Using Impala Logging</title>
+  <titlealts audience="PDF"><navtitle>Logging</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Logs"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      The Impala logs record information about:
+    </p>
+
+    <ul>
+      <li>
+        Any errors Impala encountered. If Impala experienced a serious error during startup, you must diagnose and
+        troubleshoot that problem before you can do anything further with Impala.
+      </li>
+
+      <li>
+        How Impala is configured.
+      </li>
+
+      <li>
+        Jobs Impala has completed.
+      </li>
+    </ul>
+
+    <note>
+      <p>
+        Formerly, the logs contained the query profile for each query, showing low-level details of how the work is
+        distributed among nodes and how intermediate and final results are transmitted across the network. To save
+        space, those query profiles are now stored in zlib-compressed files in
+        <filepath>/var/log/impala/profiles</filepath>. You can access them through the Impala web user interface.
+        For example, at <codeph>http://<varname>impalad-node-hostname</varname>:25000/queries</codeph>, each query
+        is followed by a <codeph>Profile</codeph> link leading to a page showing extensive analytical data for the
+        query execution.
+      </p>
+
+      <p rev="1.1.1">
+        The auditing feature introduced in Impala 1.1.1 produces a separate set of audit log files when
+        enabled. See <xref href="impala_auditing.xml#auditing"/> for details.
+      </p>
+
+      <p rev="2.2.0">
+        The lineage feature introduced in Impala 2.2.0 produces a separate lineage log file when
+        enabled. See <xref href="impala_lineage.xml#lineage"/> for details.
+      </p>
+    </note>
+
+    <p outputclass="toc inpage"/>
+
+  </conbody>
+
+  <concept id="logs_details">
+
+    <title>Locations and Names of Impala Log Files</title>
+
+    <conbody>
+
+      <ul>
+        <li>
+          By default, the log files are under the directory <filepath>/var/log/impala</filepath>.
+<!-- TK: split this task out and state CM and non-CM ways. -->
+          To change log file locations, modify the defaults file described in
+          <xref href="impala_processes.xml#processes"/>.
+        </li>
+
+        <li>
+          The significant files for the <codeph>impalad</codeph> process are <filepath>impalad.INFO</filepath>,
+          <filepath>impalad.WARNING</filepath>, and <filepath>impalad.ERROR</filepath>. You might also see a file
+          <filepath>impalad.FATAL</filepath>, although this is only present in rare conditions.
+        </li>
+
+        <li>
+          The significant files for the <codeph>statestored</codeph> process are
+          <filepath>statestored.INFO</filepath>, <filepath>statestored.WARNING</filepath>, and
+          <filepath>statestored.ERROR</filepath>. You might also see a file <filepath>statestored.FATAL</filepath>,
+          although this is only present in rare conditions.
+        </li>
+
+        <li rev="1.2">
+          The significant files for the <codeph>catalogd</codeph> process are <filepath>catalogd.INFO</filepath>,
+          <filepath>catalogd.WARNING</filepath>, and <filepath>catalogd.ERROR</filepath>. You might also see a file
+          <filepath>catalogd.FATAL</filepath>, although this is only present in rare conditions.
+        </li>
+
+        <li>
+          Examine the <codeph>.INFO</codeph> files to see configuration settings for the processes.
+        </li>
+
+        <li>
+          Examine the <codeph>.WARNING</codeph> files to see all kinds of problem information, including such
+          things as suboptimal settings and also serious runtime errors.
+        </li>
+
+        <li>
+          Examine the <codeph>.ERROR</codeph> and/or <codeph>.FATAL</codeph> files to see only the most serious
+          errors, if the processes crash, or queries fail to complete. These messages are also in the
+          <codeph>.WARNING</codeph> file.
+        </li>
+
+        <li>
+          A new set of log files is produced each time the associated daemon is restarted. These log files have
+          long names including a timestamp. The <codeph>.INFO</codeph>, <codeph>.WARNING</codeph>, and
+          <codeph>.ERROR</codeph> files are physically represented as symbolic links to the latest applicable log
+          files.
+        </li>
+
+        <li>
+          The init script for the <codeph>impala-server</codeph> service also produces a consolidated log file
+          <codeph>/var/logs/impalad/impala-server.log</codeph>, with all the same information as the
+          corresponding<codeph>.INFO</codeph>, <codeph>.WARNING</codeph>, and <codeph>.ERROR</codeph> files.
+        </li>
+
+        <li>
+          The init script for the <codeph>impala-state-store</codeph> service also produces a consolidated log file
+          <codeph>/var/logs/impalad/impala-state-store.log</codeph>, with all the same information as the
+          corresponding<codeph>.INFO</codeph>, <codeph>.WARNING</codeph>, and <codeph>.ERROR</codeph> files.
+        </li>
+      </ul>
+
+      <p>
+        Impala stores information using the <codeph>glog_v</codeph> logging system. You will see some messages
+        referring to C++ file names. Logging is affected by:
+      </p>
+
+      <ul>
+        <li>
+          The <codeph>GLOG_v</codeph> environment variable specifies which types of messages are logged. See
+          <xref href="#log_levels"/> for details.
+        </li>
+
+        <li>
+          The <codeph>-logbuflevel</codeph> startup flag for the <cmdname>impalad</cmdname> daemon specifies how
+          often the log information is written to disk. The default is 0, meaning that the log is immediately
+          flushed to disk when Impala outputs an important messages such as a warning or an error, but less
+          important messages such as informational ones are buffered in memory rather than being flushed to disk
+          immediately.
+        </li>
+
+        <li>
+          Cloudera Manager has an Impala configuration setting that sets the <codeph>-logbuflevel</codeph> startup
+          option.
+        </li>
+      </ul>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="logs_cm_noncm">
+
+    <title>Managing Impala Logs through Cloudera Manager or Manually</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Cloudera Manager"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        <ph rev="upstream">Cloudera</ph> recommends installing Impala through the Cloudera Manager administration interface. To assist with
+        troubleshooting, Cloudera Manager collects front-end and back-end logs together into a single view, and let
+        you do a search across log data for all the managed nodes rather than examining the logs on each node
+        separately. If you installed Impala using Cloudera Manager, refer to the topics on Monitoring Services
+        (<xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_dg_service_monitoring.html" scope="external" format="html">CDH 5</xref>)
+        or Logs (<xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_dg_logs.html" scope="external" format="html">CDH 5</xref>).
+      </p>
+
+      <p>
+        If you are using Impala in an environment not managed by Cloudera Manager, review Impala log files on each
+        host, when you have traced an issue back to a specific system.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="logs_rotate">
+
+    <title>Rotating Impala Logs</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Disk Storage"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        Impala periodically switches the physical files representing the current log files, after which it is safe
+        to remove the old files if they are no longer needed.
+      </p>
+
+      <p>
+        Impala can automatically remove older unneeded log files, a feature known as <term>log rotation</term>.
+<!-- Another instance of the text also used in impala_new_features.xml
+             and impala_fixed_issues.xml. (Just took out the word "new"
+             and added the reference to the starting release.)
+             At this point, a conref is definitely in the cards. -->
+      </p>
+
+      <p>
+        In Impala 2.2 and higher, the <codeph>-max_log_files</codeph> configuration option specifies how many log
+        files to keep at each severity level. You can specify an appropriate setting for each Impala-related daemon
+        (<cmdname>impalad</cmdname>, <cmdname>statestored</cmdname>, and <cmdname>catalogd</cmdname>). The default
+        value is 10, meaning that Impala preserves the latest 10 log files for each severity level
+        (<codeph>INFO</codeph>, <codeph>WARNING</codeph>, <codeph>ERROR</codeph>, and <codeph>FATAL</codeph>).
+        Impala checks to see if any old logs need to be removed based on the interval specified in the
+        <codeph>logbufsecs</codeph> setting, every 5 seconds by default.
+      </p>
+
+<!-- This extra detail only appears here. Consider if it's worth including it
+           in the conref so people don't need to follow a link just for a couple of
+           minor factoids. -->
+
+      <p>
+        A value of 0 preserves all log files, in which case you would set up set up manual log rotation using your
+        Linux tool or technique of choice. A value of 1 preserves only the very latest log file.
+      </p>
+
+      <p>
+        To set up log rotation on a system managed by Cloudera Manager 5.4.0 and higher, search for the
+        <codeph>max_log_files</codeph> option name and set the appropriate value for the <userinput>Maximum Log
+        Files</userinput> field for each Impala configuration category (Impala, Catalog Server, and StateStore).
+        Then restart the Impala service. In earlier Cloudera Manager releases, specify the
+        <codeph>-max_log_files=<varname>maximum</varname></codeph> option in the <uicontrol>Command Line Argument
+        Advanced Configuration Snippet (Safety Valve)</uicontrol> field for each Impala configuration category.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="logs_debug">
+
+    <title>Reviewing Impala Logs</title>
+
+    <conbody>
+
+      <p>
+        By default, the Impala log is stored at <codeph>/var/logs/impalad/</codeph>. The most comprehensive log,
+        showing informational, warning, and error messages, is in the file name <filepath>impalad.INFO</filepath>.
+        View log file contents by using the web interface or by examining the contents of the log file. (When you
+        examine the logs through the file system, you can troubleshoot problems by reading the
+        <filepath>impalad.WARNING</filepath> and/or <filepath>impalad.ERROR</filepath> files, which contain the
+        subsets of messages indicating potential problems.)
+      </p>
+
+      <p>
+        On a machine named <codeph>impala.example.com</codeph> with default settings, you could view the Impala
+        logs on that machine by using a browser to access <codeph>http://impala.example.com:25000/logs</codeph>.
+      </p>
+
+      <note>
+        <p>
+          The web interface limits the amount of logging information displayed. To view every log entry, access the
+          log files directly through the file system.
+        </p>
+      </note>
+
+      <p>
+        You can view the contents of the <codeph>impalad.INFO</codeph> log file in the file system. With the
+        default configuration settings, the start of the log file appears as follows:
+      </p>
+
+<codeblock>[user@example impalad]$ pwd
+/var/log/impalad
+[user@example impalad]$ more impalad.INFO
+Log file created at: 2013/01/07 08:42:12
+Running on machine: impala.example.com
+Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
+I0107 08:42:12.292155 14876 daemon.cc:34] impalad version 0.4 RELEASE (build 9d7fadca0461ab40b9e9df8cdb47107ec6b27cff)
+Built on Fri, 21 Dec 2012 12:55:19 PST
+I0107 08:42:12.292484 14876 daemon.cc:35] Using hostname: impala.example.com
+I0107 08:42:12.292706 14876 logging.cc:76] Flags (see also /varz are on debug webserver):
+--dump_ir=false
+--module_output=
+--be_port=22000
+--classpath=
+--hostname=impala.example.com</codeblock>
+
+      <note>
+        The preceding example shows only a small part of the log file. Impala log files are often several megabytes
+        in size.
+      </note>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="log_format">
+
+    <title>Understanding Impala Log Contents</title>
+
+    <conbody>
+
+      <p>
+        The logs store information about Impala startup options. This information appears once for each time Impala
+        is started and may include:
+      </p>
+
+      <ul>
+        <li>
+          Machine name.
+        </li>
+
+        <li>
+          Impala version number.
+        </li>
+
+        <li>
+          Flags used to start Impala.
+        </li>
+
+        <li>
+          CPU information.
+        </li>
+
+        <li>
+          The number of available disks.
+        </li>
+      </ul>
+
+      <p>
+        There is information about each job Impala has run. Because each Impala job creates an additional set of
+        data about queries, the amount of job specific data may be very large. Logs may contained detailed
+        information on jobs. These detailed log entries may include:
+      </p>
+
+      <ul>
+        <li>
+          The composition of the query.
+        </li>
+
+        <li>
+          The degree of data locality.
+        </li>
+
+        <li>
+          Statistics on data throughput and response times.
+        </li>
+      </ul>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="log_levels">
+
+    <title>Setting Logging Levels</title>
+
+    <conbody>
+
+      <p>
+        Impala uses the GLOG system, which supports three logging levels. You can adjust the logging levels using
+        the Cloudera Manager Admin Console. You can adjust logging levels without going through the Cloudera
+        Manager Admin Console by exporting variable settings. To change logging settings manually, use a command
+        similar to the following on each node before starting <codeph>impalad</codeph>:
+      </p>
+
+<codeblock>export GLOG_v=1</codeblock>
+
+      <note>
+        For performance reasons, Cloudera highly recommends not enabling the most verbose logging level of 3.
+      </note>
+
+      <p>
+        For more information on how to configure GLOG, including how to set variable logging levels for different
+        system components, see
+        <xref href="http://google-glog.googlecode.com/svn/trunk/doc/glog.html" scope="external" format="html">How
+        To Use Google Logging Library (glog)</xref>.
+      </p>
+
+      <section id="loglevels_details">
+
+        <title>Understanding What is Logged at Different Logging Levels</title>
+
+        <p>
+          As logging levels increase, the categories of information logged are cumulative. For example, GLOG_v=2
+          records everything GLOG_v=1 records, as well as additional information.
+        </p>
+
+        <p>
+          Increasing logging levels imposes performance overhead and increases log size. <ph rev="upstream">Cloudera</ph> recommends using
+          GLOG_v=1 for most cases: this level has minimal performance impact but still captures useful
+          troubleshooting information.
+        </p>
+
+        <p>
+          Additional information logged at each level is as follows:
+        </p>
+
+        <ul>
+          <li>
+            GLOG_v=1 - The default level. Logs information about each connection and query that is initiated to an
+            <codeph>impalad</codeph> instance, including runtime profiles.
+          </li>
+
+          <li>
+            GLOG_v=2 - Everything from the previous level plus information for each RPC initiated. This level also
+            records query execution progress information, including details on each file that is read.
+          </li>
+
+          <li>
+            GLOG_v=3 - Everything from the previous level plus logging of every row that is read. This level is
+            only applicable for the most serious troubleshooting and tuning scenarios, because it can produce
+            exceptionally large and detailed log files, potentially leading to its own set of performance and
+            capacity problems.
+          </li>
+        </ul>
+
+      </section>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="redaction" rev="2.2.0">
+
+    <title>Redacting Sensitive Information from Impala Log Files</title>
+    <prolog>
+      <metadata>
+        <data name="Category" value="Redaction"/>
+      </metadata>
+    </prolog>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">redaction</indexterm>
+        <term>Log redaction</term> is a security feature that prevents sensitive information from being displayed in
+        locations used by administrators for monitoring and troubleshooting, such as log files, the Cloudera Manager
+        user interface, and the Impala debug web user interface. You configure regular expressions that match
+        sensitive types of information processed by your system, such as credit card numbers or tax IDs, and literals
+        matching these patterns are obfuscated wherever they would normally be recorded in log files or displayed in
+        administration or debugging user interfaces.
+      </p>
+
+      <p>
+        In a security context, the log redaction feature is complementary to the Sentry authorization framework.
+        Sentry prevents unauthorized users from being able to directly access table data. Redaction prevents
+        administrators or support personnel from seeing the smaller amounts of sensitive or personally identifying
+        information (PII) that might appear in queries issued by those authorized users.
+      </p>
+
+      <p>
+        See
+        <xref audience="integrated" href="sg_redaction.xml#log_redact"/><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/sg_redaction.html" scope="external" format="html"/>
+        for details about how to enable this feature and set
+        up the regular expressions to detect and redact sensitive information within SQL statement text.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_map.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_map.xml b/docs/topics/impala_map.xml
new file mode 100644
index 0000000..a20c356
--- /dev/null
+++ b/docs/topics/impala_map.xml
@@ -0,0 +1,267 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+  <concept id="map">
+
+    <title>MAP Complex Type (<keyword keyref="impala23"/> or higher only)</title>
+
+    <prolog>
+      <metadata>
+        <data name="Category" value="Impala"/>
+        <data name="Category" value="Impala Data Types"/>
+        <data name="Category" value="SQL"/>
+        <data name="Category" value="Developers"/>
+        <data name="Category" value="Data Analysts"/>
+      </metadata>
+    </prolog>
+
+    <conbody>
+
+      <p>
+        A complex data type representing an arbitrary set of key-value pairs.
+        The key part is a scalar type, while the value part can be a scalar or
+        another complex type (<codeph>ARRAY</codeph>, <codeph>STRUCT</codeph>,
+        or <codeph>MAP</codeph>).
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock><varname>column_name</varname> MAP &lt; <varname>primitive_type</varname>, <varname>type</varname> &gt;
+
+type ::= <varname>primitive_type</varname> | <varname>complex_type</varname>
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_combo"/>
+
+      <p>
+        The <codeph>MAP</codeph> complex data type represents a set of key-value pairs.
+        Each element of the map is indexed by a primitive type such as <codeph>BIGINT</codeph> or
+        <codeph>STRING</codeph>, letting you define sequences that are not continuous or categories with arbitrary names.
+        You might find it convenient for modelling data produced in other languages, such as a
+        Python dictionary or Java HashMap, where a single scalar value serves as the lookup key.
+      </p>
+
+      <p>
+        In a big data context, the keys in a map column might represent a numeric sequence of events during a
+        manufacturing process, or <codeph>TIMESTAMP</codeph> values corresponding to sensor observations.
+        The map itself is inherently unordered, so you choose whether to make the key values significant
+        (such as a recorded <codeph>TIMESTAMP</codeph>) or synthetic (such as a random global universal ID).
+      </p>
+
+      <note>
+        Behind the scenes, the <codeph>MAP</codeph> type is implemented in a similar way as the
+        <codeph>ARRAY</codeph> type. Impala does not enforce any uniqueness constraint on the
+        <codeph>KEY</codeph> values, and the <codeph>KEY</codeph> values are processed by
+        looping through the elements of the <codeph>MAP</codeph> rather than by a constant-time lookup.
+        Therefore, this type is primarily for ease of understanding when importing data and
+        algorithms from non-SQL contexts, rather than optimizing the performance of key lookups.
+      </note>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_describe"/>
+
+      <p conref="../shared/impala_common.xml#common/added_in_230"/>
+
+      <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+      <ul conref="../shared/impala_common.xml#common/complex_types_restrictions">
+        <li/>
+      </ul>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <note conref="../shared/impala_common.xml#common/complex_type_schema_pointer"/>
+
+      <p>
+        The following example shows a table with various kinds of <codeph>MAP</codeph> columns,
+        both at the top level and nested within other complex types.
+        Each row represents information about a specific country, with complex type fields
+        of various levels of nesting to represent different information associated
+        with the country: factual measurements such as area and population,
+        notable people in different categories, geographic features such as
+        cities, points of interest within each city, and mountains with associated facts.
+        Practice the <codeph>CREATE TABLE</codeph> and query notation for complex type columns
+        using empty tables, until you can visualize a complex data structure and construct corresponding SQL statements reliably.
+      </p>
+
+<codeblock><![CDATA[create TABLE map_demo
+(
+  country_id BIGINT,
+
+-- Numeric facts about each country, looked up by name.
+-- For example, 'Area':1000, 'Population':999999.
+-- Using a MAP instead of a STRUCT because there could be
+-- a different set of facts for each country.
+  metrics MAP <STRING, BIGINT>,
+
+-- MAP whose value part is an ARRAY.
+-- For example, the key 'Famous Politicians' could represent an array of 10 elements,
+-- while the key 'Famous Actors' could represent an array of 20 elements.
+  notables MAP <STRING, ARRAY <STRING>>,
+
+-- MAP that is a field within a STRUCT.
+-- (The STRUCT is inside another ARRAY, because it is rare
+-- for a STRUCT to be a top-level column.)
+-- For example, city #1 might have points of interest with key 'Zoo',
+-- representing an array of 3 different zoos.
+-- City #2 might have completely different kinds of points of interest.
+-- Because the set of field names is potentially large, and most entries could be blank,
+-- a MAP makes more sense than a STRUCT to represent such a sparse data structure.
+  cities ARRAY < STRUCT <
+    name: STRING,
+    points_of_interest: MAP <STRING, ARRAY <STRING>>
+  >>,
+
+-- MAP that is an element within an ARRAY. The MAP is inside a STRUCT field to associate
+-- the mountain name with all the facts about the mountain.
+-- The "key" of the map (the first STRING field) represents the name of some fact whose value
+-- can be expressed as an integer, such as 'Height', 'Year First Climbed', and so on.
+  mountains ARRAY < STRUCT < name: STRING, facts: MAP <STRING, INT > > >
+)
+STORED AS PARQUET;
+]]>
+</codeblock>
+
+<codeblock><![CDATA[DESCRIBE map_demo;
++------------+------------------------------------------------+
+| name       | type                                           |
++------------+------------------------------------------------+
+| country_id | bigint                                         |
+| metrics    | map<string,bigint>                             |
+| notables   | map<string,array<string>>                      |
+| cities     | array<struct<                                  |
+|            |   name:string,                                 |
+|            |   points_of_interest:map<string,array<string>> |
+|            | >>                                             |
+| mountains  | array<struct<                                  |
+|            |   name:string,                                 |
+|            |   facts:map<string,int>                        |
+|            | >>                                             |
++------------+------------------------------------------------+
+
+DESCRIBE map_demo.metrics;
++-------+--------+
+| name  | type   |
++-------+--------+
+| key   | string |
+| value | bigint |
++-------+--------+
+
+DESCRIBE map_demo.notables;
++-------+---------------+
+| name  | type          |
++-------+---------------+
+| key   | string        |
+| value | array<string> |
++-------+---------------+
+
+DESCRIBE map_demo.notables.value;
++------+--------+
+| name | type   |
++------+--------+
+| item | string |
+| pos  | bigint |
++------+--------+
+
+DESCRIBE map_demo.cities;
++------+------------------------------------------------+
+| name | type                                           |
++------+------------------------------------------------+
+| item | struct<                                        |
+|      |   name:string,                                 |
+|      |   points_of_interest:map<string,array<string>> |
+|      | >                                              |
+| pos  | bigint                                         |
++------+------------------------------------------------+
+
+DESCRIBE map_demo.cities.item.points_of_interest;
++-------+---------------+
+| name  | type          |
++-------+---------------+
+| key   | string        |
+| value | array<string> |
++-------+---------------+
+
+DESCRIBE map_demo.cities.item.points_of_interest.value;
++------+--------+
+| name | type   |
++------+--------+
+| item | string |
+| pos  | bigint |
++------+--------+
+
+DESCRIBE map_demo.mountains;
++------+-------------------------+
+| name | type                    |
++------+-------------------------+
+| item | struct<                 |
+|      |   name:string,          |
+|      |   facts:map<string,int> |
+|      | >                       |
+| pos  | bigint                  |
++------+-------------------------+
+
+DESCRIBE map_demo.mountains.item.facts;
++-------+--------+
+| name  | type   |
++-------+--------+
+| key   | string |
+| value | int    |
++-------+--------+
+]]>
+</codeblock>
+
+      <p>
+        The following example shows a table that uses a variety of data types for the <codeph>MAP</codeph>
+        <q>key</q> field. Typically, you use <codeph>BIGINT</codeph> or <codeph>STRING</codeph> to use
+        numeric or character-based keys without worrying about exceeding any size or length constraints.
+      </p>
+
+<codeblock><![CDATA[CREATE TABLE map_demo_obscure
+(
+  id BIGINT,
+  m1 MAP <INT, INT>,
+  m2 MAP <SMALLINT, INT>,
+  m3 MAP <TINYINT, INT>,
+  m4 MAP <TIMESTAMP, INT>,
+  m5 MAP <BOOLEAN, INT>,
+  m6 MAP <CHAR(5), INT>,
+  m7 MAP <VARCHAR(25), INT>,
+  m8 MAP <FLOAT, INT>,
+  m9 MAP <DOUBLE, INT>,
+  m10 MAP <DECIMAL(12,2), INT>
+)
+STORED AS PARQUET;
+]]>
+</codeblock>
+
+<codeblock>CREATE TABLE celebrities (name STRING, birth_year MAP &lt; STRING, SMALLINT &gt;) STORED AS PARQUET;
+-- A typical row might represent values with 2 different birth years, such as:
+-- ("Joe Movie Star", { "real": 1972, "claimed": 1977 })
+
+CREATE TABLE countries (name STRING, famous_leaders MAP &lt; INT, STRING &gt;) STORED AS PARQUET;
+-- A typical row might represent values with different leaders, with key values corresponding to their numeric sequence, such as:
+-- ("United States", { 1: "George Washington", 3: "Thomas Jefferson", 16: "Abraham Lincoln" })
+
+CREATE TABLE airlines (name STRING, special_meals MAP &lt; STRING, MAP &lt; STRING, STRING &gt; &gt;) STORED AS PARQUET;
+-- A typical row might represent values with multiple kinds of meals, each with several components:
+-- ("Elegant Airlines",
+--   {
+--     "vegetarian": { "breakfast": "pancakes", "snack": "cookies", "dinner": "rice pilaf" },
+--     "gluten free": { "breakfast": "oatmeal", "snack": "fruit", "dinner": "chicken" }
+--   } )
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_complex_types.xml#complex_types"/>,
+        <xref href="impala_array.xml#array"/>,
+        <xref href="impala_struct.xml#struct"/>
+        <!-- <xref href="impala_map.xml#map"/> -->
+      </p>
+
+    </conbody>
+
+  </concept>
+
+

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_math_functions.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_math_functions.xml b/docs/topics/impala_math_functions.xml
new file mode 100644
index 0000000..c82a29b
--- /dev/null
+++ b/docs/topics/impala_math_functions.xml
@@ -0,0 +1,1389 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="math_functions">
+
+  <title>Impala Mathematical Functions</title>
+  <titlealts audience="PDF"><navtitle>Mathematical Functions</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Querying"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Mathematical functions, or arithmetic functions, perform numeric calculations that are typically more complex
+      than basic addition, subtraction, multiplication, and division. For example, these functions include
+      trigonometric, logarithmic, and base conversion operations.
+    </p>
+
+    <note>
+      In Impala, exponentiation uses the <codeph>pow()</codeph> function rather than an exponentiation operator
+      such as <codeph>**</codeph>.
+    </note>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      The mathematical functions operate mainly on these data types: <xref href="impala_int.xml#int"/>,
+      <xref href="impala_bigint.xml#bigint"/>, <xref href="impala_smallint.xml#smallint"/>,
+      <xref href="impala_tinyint.xml#tinyint"/>, <xref href="impala_double.xml#double"/>,
+      <xref href="impala_float.xml#float"/>, and <xref href="impala_decimal.xml#decimal"/>. For the operators that
+      perform the standard operations such as addition, subtraction, multiplication, and division, see
+      <xref href="impala_operators.xml#arithmetic_operators"/>.
+    </p>
+
+    <p>
+      Functions that perform bitwise operations are explained in <xref href="impala_bit_functions.xml#bit_functions"/>.
+    </p>
+
+    <p>
+      <b>Function reference:</b>
+    </p>
+
+    <p>
+      Impala supports the following mathematical functions:
+    </p>
+
+    <dl>
+      <dlentry rev="1.4.0" id="abs">
+
+        <dt rev="1.4.0 2.0.1">
+          <codeph>abs(numeric_type a)</codeph>
+<!-- <codeph>abs(double a), abs(decimal(p,s) a)</codeph> -->
+        </dt>
+
+        <dd rev="1.4.0">
+          <indexterm audience="Cloudera">abs() function</indexterm>
+          <b>Purpose:</b> Returns the absolute value of the argument.
+          <p rev="2.0.1" conref="../shared/impala_common.xml#common/return_type_same"/>
+          <p>
+            <b>Usage notes:</b> Use this function to ensure all return values are positive. This is different than
+            the <codeph>positive()</codeph> function, which returns its argument unchanged (even if the argument
+            was negative).
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="acos">
+
+        <dt>
+          <codeph>acos(double a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">acos() function</indexterm>
+          <b>Purpose:</b> Returns the arccosine of the argument.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="asin">
+
+        <dt>
+          <codeph>asin(double a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">asin() function</indexterm>
+          <b>Purpose:</b> Returns the arcsine of the argument.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="atan">
+
+        <dt>
+          <codeph>atan(double a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">atan() function</indexterm>
+          <b>Purpose:</b> Returns the arctangent of the argument.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="atan2" rev="2.3.0 IMPALA-1771">
+
+        <dt rev="2.3.0 IMPALA-1771">
+          <codeph>atan2(double a, double b)</codeph>
+        </dt>
+
+        <dd rev="2.3.0 IMPALA-1771">
+          <indexterm audience="Cloudera">atan2() function</indexterm>
+          <b>Purpose:</b> Returns the arctangent of the two arguments, with the signs of the arguments used to determine the
+          quadrant of the result.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="bin">
+
+        <dt>
+          <codeph>bin(bigint a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">bin() function</indexterm>
+          <b>Purpose:</b> Returns the binary representation of an integer value, that is, a string of 0 and 1
+          digits.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.4.0" id="ceil">
+
+        <dt rev="1.4.0">
+          <codeph>ceil(double a)</codeph>,
+          <codeph>ceil(decimal(p,s) a)</codeph>,
+          <codeph id="ceiling">ceiling(double a)</codeph>,
+          <codeph>ceiling(decimal(p,s) a)</codeph>,
+          <codeph id="dceil" rev="2.3.0">dceil(double a)</codeph>,
+          <codeph rev="2.3.0">dceil(decimal(p,s) a)</codeph>
+        </dt>
+
+        <dd rev="1.4.0">
+          <indexterm audience="Cloudera">ceil() function</indexterm>
+          <b>Purpose:</b> Returns the smallest integer that is greater than or equal to the argument.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph> or <codeph>decimal(p,s)</codeph> depending on the type of the
+            input argument
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="conv">
+
+        <dt>
+          <codeph>conv(bigint num, int from_base, int to_base), conv(string num, int from_base, int
+          to_base)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">conv() function</indexterm>
+          <b>Purpose:</b> Returns a string representation of an integer value in a particular base. The input value
+          can be a string, for example to convert a hexadecimal number such as <codeph>fce2</codeph> to decimal. To
+          use the return value as a number (for example, when converting to base 10), use <codeph>CAST()</codeph>
+          to convert to the appropriate type.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="cos">
+
+        <dt>
+          <codeph>cos(double a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">cos() function</indexterm>
+          <b>Purpose:</b> Returns the cosine of the argument.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="cosh" rev="2.3.0 IMPALA-1771">
+
+        <dt rev="2.3.0 IMPALA-1771">
+          <codeph>cosh(double a)</codeph>
+        </dt>
+
+        <dd rev="2.3.0 IMPALA-1771">
+          <indexterm audience="Cloudera">cosh() function</indexterm>
+          <b>Purpose:</b> Returns the hyperbolic cosine of the argument.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="cot" rev="2.3.0 IMPALA-1771">
+
+        <dt rev="2.3.0 IMPALA-1771">
+          <codeph>cot(double a)</codeph>
+        </dt>
+
+        <dd rev="2.3.0 IMPALA-1771">
+          <indexterm audience="Cloudera">cot() function</indexterm>
+          <b>Purpose:</b> Returns the cotangent of the argument.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="degrees">
+
+        <dt>
+          <codeph>degrees(double a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">degrees() function</indexterm>
+          <b>Purpose:</b> Converts argument value from radians to degrees.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="e">
+
+        <dt>
+          <codeph>e()</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">e() function</indexterm>
+          <b>Purpose:</b> Returns the
+          <xref href="https://en.wikipedia.org/wiki/E_(mathematical_constant" scope="external" format="html">mathematical
+          constant e</xref>.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="exp">
+
+        <dt>
+          <codeph>exp(double a)</codeph>,
+          <codeph rev="2.3.0" id="dexp">dexp(double a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">exp() function</indexterm>
+          <b>Purpose:</b> Returns the
+          <xref href="https://en.wikipedia.org/wiki/E_(mathematical_constant" scope="external" format="html">mathematical
+          constant e</xref> raised to the power of the argument.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="2.3.0" id="factorial">
+
+        <dt rev="2.3.0">
+          <codeph>factorial(integer_type a)</codeph>
+        </dt>
+        <dd rev="2.3.0">
+          <indexterm audience="Cloudera">factorial() function</indexterm>
+          <b>Purpose:</b> Computes the <xref href="https://en.wikipedia.org/wiki/Factorial" scope="external" format="html">factorial</xref> of an integer value.
+          It works with any integer type.
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+          <p>
+            <b>Usage notes:</b> You can use either the <codeph>factorial()</codeph> function or the <codeph>!</codeph> operator.
+            The factorial of 0 is 1. Likewise, the <codeph>factorial()</codeph> function returns 1 for any negative value.
+            The maximum positive value for the input argument is 20; a value of 21 or greater overflows the
+            range for a <codeph>BIGINT</codeph> and causes an error.
+          </p>
+          <p>
+            <b>Return type:</b> <codeph>bigint</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/added_in_230"/>
+<codeblock>select factorial(5);
++--------------+
+| factorial(5) |
++--------------+
+| 120          |
++--------------+
+
+select 5!;
++-----+
+| 5!  |
++-----+
+| 120 |
++-----+
+
+select factorial(0);
++--------------+
+| factorial(0) |
++--------------+
+| 1            |
++--------------+
+
+select factorial(-100);
++-----------------+
+| factorial(-100) |
++-----------------+
+| 1               |
++-----------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="floor">
+
+        <dt>
+          <codeph>floor(double a)</codeph>,
+          <codeph>floor(decimal(p,s) a)</codeph>,
+          <codeph rev="2.3.0" id="dfloor">dfloor(double a)</codeph>,
+          <codeph rev="2.3.0">dfloor(decimal(p,s) a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">floor() function</indexterm>
+          <b>Purpose:</b> Returns the largest integer that is less than or equal to the argument.
+          <p>
+            <b>Return type:</b> <codeph>bigint</codeph> or <codeph>decimal(p,s)</codeph> depending on the type of
+            the input argument
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="fmod">
+
+        <dt>
+          <codeph>fmod(double a, double b), fmod(float a, float b)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">fmod() function</indexterm>
+          <b>Purpose:</b> Returns the modulus of a floating-point number. Equivalent to the <codeph>%</codeph> arithmetic operator.
+          <p>
+            <b>Return type:</b> <codeph>float</codeph> or <codeph>double</codeph>, depending on type of arguments
+          </p>
+          <p conref="../shared/impala_common.xml#common/added_in_111"/>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            Because this function operates on <codeph>DOUBLE</codeph> or <codeph>FLOAT</codeph>
+            values, it is subject to potential rounding errors for values that cannot be
+            represented precisely. Prefer to use whole numbers, or values that you know
+            can be represented precisely by the <codeph>DOUBLE</codeph> or <codeph>FLOAT</codeph>
+            types.
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples show equivalent operations with the <codeph>fmod()</codeph>
+            function and the <codeph>%</codeph> arithmetic operator, for values not subject
+            to any rounding error.
+          </p>
+<codeblock>select fmod(10,3);
++-------------+
+| fmod(10, 3) |
++-------------+
+| 1           |
++-------------+
+
+select fmod(5.5,2);
++--------------+
+| fmod(5.5, 2) |
++--------------+
+| 1.5          |
++--------------+
+
+select 10 % 3;
++--------+
+| 10 % 3 |
++--------+
+| 1      |
++--------+
+
+select 5.5 % 2;
++---------+
+| 5.5 % 2 |
++---------+
+| 1.5     |
++---------+
+</codeblock>
+          <p>
+            The following examples show operations with the <codeph>fmod()</codeph>
+            function for values that cannot be represented precisely by the
+            <codeph>DOUBLE</codeph> or <codeph>FLOAT</codeph> types, and thus are
+            subject to rounding error. <codeph>fmod(9.9,3.0)</codeph> returns a value
+            slightly different than the expected 0.9 because of rounding.
+            <codeph>fmod(9.9,3.3)</codeph> returns a value quite different from
+            the expected value of 0 because of rounding error during intermediate
+            calculations.
+          </p>
+<codeblock>select fmod(9.9,3.0);
++--------------------+
+| fmod(9.9, 3.0)     |
++--------------------+
+| 0.8999996185302734 |
++--------------------+
+
+select fmod(9.9,3.3);
++-------------------+
+| fmod(9.9, 3.3)    |
++-------------------+
+| 3.299999713897705 |
++-------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.2.2" id="fnv_hash">
+
+        <dt rev="1.2.2">
+          <codeph>fnv_hash(type v)</codeph>,
+        </dt>
+
+        <dd rev="1.2.2">
+          <indexterm audience="Cloudera">fnv_hash() function</indexterm>
+          <b>Purpose:</b> Returns a consistent 64-bit value derived from the input argument, for convenience of
+          implementing hashing logic in an application.
+          <p>
+            <b>Return type:</b> <codeph>BIGINT</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            You might use the return value in an application where you perform load balancing, bucketing, or some
+            other technique to divide processing or storage.
+          </p>
+          <p>
+            Because the result can be any 64-bit value, to restrict the value to a particular range, you can use an
+            expression that includes the <codeph>ABS()</codeph> function and the <codeph>%</codeph> (modulo)
+            operator. For example, to produce a hash value in the range 0-9, you could use the expression
+            <codeph>ABS(FNV_HASH(x)) % 10</codeph>.
+          </p>
+          <p>
+            This function implements the same algorithm that Impala uses internally for hashing, on systems where
+            the CRC32 instructions are not available.
+          </p>
+          <p>
+            This function implements the
+            <xref href="http://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function" scope="external" format="html">Fowler\u2013Noll\u2013Vo
+            hash function</xref>, in particular the FNV-1a variation. This is not a perfect hash function: some
+            combinations of values could produce the same result value. It is not suitable for cryptographic use.
+          </p>
+          <p>
+            Similar input values of different types could produce different hash values, for example the same
+            numeric value represented as <codeph>SMALLINT</codeph> or <codeph>BIGINT</codeph>,
+            <codeph>FLOAT</codeph> or <codeph>DOUBLE</codeph>, or <codeph>DECIMAL(5,2)</codeph> or
+            <codeph>DECIMAL(20,5)</codeph>.
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>[localhost:21000] &gt; create table h (x int, s string);
+[localhost:21000] &gt; insert into h values (0, 'hello'), (1,'world'), (1234567890,'antidisestablishmentarianism');
+[localhost:21000] &gt; select x, fnv_hash(x) from h;
++------------+----------------------+
+| x          | fnv_hash(x)          |
++------------+----------------------+
+| 0          | -2611523532599129963 |
+| 1          | 4307505193096137732  |
+| 1234567890 | 3614724209955230832  |
++------------+----------------------+
+[localhost:21000] &gt; select s, fnv_hash(s) from h;
++------------------------------+---------------------+
+| s                            | fnv_hash(s)         |
++------------------------------+---------------------+
+| hello                        | 6414202926103426347 |
+| world                        | 6535280128821139475 |
+| antidisestablishmentarianism | -209330013948433970 |
++------------------------------+---------------------+
+[localhost:21000] &gt; select s, abs(fnv_hash(s)) % 10 from h;
++------------------------------+-------------------------+
+| s                            | abs(fnv_hash(s)) % 10.0 |
++------------------------------+-------------------------+
+| hello                        | 8                       |
+| world                        | 6                       |
+| antidisestablishmentarianism | 4                       |
++------------------------------+-------------------------+</codeblock>
+          <p>
+            For short argument values, the high-order bits of the result have relatively low entropy:
+          </p>
+<codeblock>[localhost:21000] &gt; create table b (x boolean);
+[localhost:21000] &gt; insert into b values (true), (true), (false), (false);
+[localhost:21000] &gt; select x, fnv_hash(x) from b;
++-------+---------------------+
+| x     | fnv_hash(x)         |
++-------+---------------------+
+| true  | 2062020650953872396 |
+| true  | 2062020650953872396 |
+| false | 2062021750465500607 |
+| false | 2062021750465500607 |
++-------+---------------------+</codeblock>
+          <p>
+            <b>Added in:</b> Impala 1.2.2
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.4.0" id="greatest">
+
+        <dt rev="1.4.0">
+          <codeph>greatest(bigint a[, bigint b ...])</codeph>, <codeph>greatest(double a[, double b ...])</codeph>,
+          <codeph>greatest(decimal(p,s) a[, decimal(p,s) b ...])</codeph>, <codeph>greatest(string a[, string b
+          ...])</codeph>, <codeph>greatest(timestamp a[, timestamp b ...])</codeph>
+        </dt>
+
+        <dd rev="1.4.0">
+          <indexterm audience="Cloudera">greatest() function</indexterm>
+          <b>Purpose:</b> Returns the largest value from a list of expressions.
+          <p conref="../shared/impala_common.xml#common/return_same_type"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="hex">
+
+        <dt>
+          <codeph>hex(bigint a), hex(string a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">hex() function</indexterm>
+          <b>Purpose:</b> Returns the hexadecimal representation of an integer value, or of the characters in a
+          string.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.4.0" id="is_inf">
+
+        <dt rev="1.4.0">
+          <codeph>is_inf(double a)</codeph>,
+        </dt>
+
+        <dd rev="1.4.0">
+          <indexterm audience="Cloudera">is_inf() function</indexterm>
+          <b>Purpose:</b> Tests whether a value is equal to the special value <q>inf</q>, signifying infinity.
+          <p>
+            <b>Return type:</b> <codeph>boolean</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p conref="../shared/impala_common.xml#common/infinity_and_nan"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.4.0" id="is_nan">
+
+        <dt rev="1.4.0">
+          <codeph>is_nan(double a)</codeph>,
+        </dt>
+
+        <dd rev="1.4.0">
+          <indexterm audience="Cloudera">is_nan() function</indexterm>
+          <b>Purpose:</b> Tests whether a value is equal to the special value <q>NaN</q>, signifying <q>not a
+          number</q>.
+          <p>
+            <b>Return type:</b> <codeph>boolean</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p conref="../shared/impala_common.xml#common/infinity_and_nan"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.4.0" id="least">
+
+        <dt rev="1.4.0">
+          <codeph>least(bigint a[, bigint b ...])</codeph>, <codeph>least(double a[, double b ...])</codeph>,
+          <codeph>least(decimal(p,s) a[, decimal(p,s) b ...])</codeph>, <codeph>least(string a[, string b
+          ...])</codeph>, <codeph>least(timestamp a[, timestamp b ...])</codeph>
+        </dt>
+
+        <dd rev="1.4.0">
+          <indexterm audience="Cloudera">least() function</indexterm>
+          <b>Purpose:</b> Returns the smallest value from a list of expressions.
+          <p conref="../shared/impala_common.xml#common/return_same_type"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="ln">
+
+        <dt>
+          <codeph>ln(double a)</codeph>,
+          <codeph rev="2.3.0" id="dlog1">dlog1(double a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">ln() function</indexterm>
+          <indexterm audience="Cloudera">dlog1() function</indexterm>
+          <b>Purpose:</b> Returns the
+          <xref href="https://en.wikipedia.org/wiki/Natural_logarithm" scope="external" format="html">natural
+          logarithm</xref> of the argument.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="log">
+
+        <dt>
+          <codeph>log(double base, double a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">log() function</indexterm>
+          <b>Purpose:</b> Returns the logarithm of the second argument to the specified base.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="log10">
+
+        <dt>
+          <codeph>log10(double a)</codeph>,
+          <codeph rev="2.3.0" id="dlog10">dlog10(double a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">log10() function</indexterm>
+          <indexterm audience="Cloudera">dlog10() function</indexterm>
+          <b>Purpose:</b> Returns the logarithm of the argument to the base 10.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="log2">
+
+        <dt>
+          <codeph>log2(double a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">log2() function</indexterm>
+          <b>Purpose:</b> Returns the logarithm of the argument to the base 2.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.4.0" id="max_int">
+
+        <dt rev="1.4.0">
+          <codeph>max_int(), <ph id="max_tinyint">max_tinyint()</ph>, <ph id="max_smallint">max_smallint()</ph>,
+          <ph id="max_bigint">max_bigint()</ph></codeph>
+        </dt>
+
+        <dd rev="1.4.0">
+          <indexterm audience="Cloudera">max_int() function</indexterm>
+          <indexterm audience="Cloudera">max_tinyint() function</indexterm>
+          <indexterm audience="Cloudera">max_smallint() function</indexterm>
+          <indexterm audience="Cloudera">max_bigint() function</indexterm>
+          <b>Purpose:</b> Returns the largest value of the associated integral type.
+          <p>
+            <b>Return type:</b> The same as the integral type being checked.
+          </p>
+          <p>
+<!-- Repeated usage text between max_ and min_ functions, could turn into a conref. -->
+            <b>Usage notes:</b> Use the corresponding <codeph>min_</codeph> and <codeph>max_</codeph> functions to
+            check if all values in a column are within the allowed range, before copying data or altering column
+            definitions. If not, switch to the next higher integral type or to a <codeph>DECIMAL</codeph> with
+            sufficient precision.
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.4.0" id="min_int">
+
+        <dt rev="1.4.0">
+          <codeph>min_int(), <ph id="min_tinyint">min_tinyint()</ph>, <ph id="min_smallint">min_smallint()</ph>,
+          <ph id="min_bigint">min_bigint()</ph></codeph>
+        </dt>
+
+        <dd rev="1.4.0">
+          <indexterm audience="Cloudera">min_int() function</indexterm>
+          <indexterm audience="Cloudera">min_tinyint() function</indexterm>
+          <indexterm audience="Cloudera">min_smallint() function</indexterm>
+          <indexterm audience="Cloudera">min_bigint() function</indexterm>
+          <b>Purpose:</b> Returns the smallest value of the associated integral type (a negative number).
+          <p>
+            <b>Return type:</b> The same as the integral type being checked.
+          </p>
+          <p>
+            <b>Usage notes:</b> Use the corresponding <codeph>min_</codeph> and <codeph>max_</codeph> functions to
+            check if all values in a column are within the allowed range, before copying data or altering column
+            definitions. If not, switch to the next higher integral type or to a <codeph>DECIMAL</codeph> with
+            sufficient precision.
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="mod" rev="2.2.0">
+
+        <dt rev="2.2.0">
+          <codeph>mod(<varname>numeric_type</varname> a, <varname>same_type</varname> b)</codeph>
+        </dt>
+
+        <dd rev="2.2.0">
+          <indexterm audience="Cloudera">mod() function</indexterm>
+          <b>Purpose:</b> Returns the modulus of a number. Equivalent to the <codeph>%</codeph> arithmetic operator.
+          Works with any size integer type, any size floating-point type, and <codeph>DECIMAL</codeph>
+          with any precision and scale.
+          <p conref="../shared/impala_common.xml#common/return_type_same"/>
+          <p conref="../shared/impala_common.xml#common/added_in_220"/>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            Because this function works with <codeph>DECIMAL</codeph> values, prefer it over <codeph>fmod()</codeph>
+            when working with fractional values. It is not subject to the rounding errors that make
+            <codeph>fmod()</codeph> problematic with floating-point numbers.
+            The <codeph>%</codeph> arithmetic operator now uses the <codeph>mod()</codeph> function
+            in cases where its arguments can be interpreted as <codeph>DECIMAL</codeph> values,
+            increasing the accuracy of that operator.
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples show how the <codeph>mod()</codeph> function works for
+            whole numbers and fractional values, and how the <codeph>%</codeph> operator
+            works the same way. In the case of <codeph>mod(9.9,3)</codeph>,
+            the type conversion for the second argument results in the first argument
+            being interpreted as <codeph>DOUBLE</codeph>, so to produce an accurate
+            <codeph>DECIMAL</codeph> result requires casting the second argument
+            or writing it as a <codeph>DECIMAL</codeph> literal, 3.0.
+          </p>
+<codeblock>select mod(10,3);
++-------------+
+| fmod(10, 3) |
++-------------+
+| 1           |
++-------------+
+
+select mod(5.5,2);
++--------------+
+| fmod(5.5, 2) |
++--------------+
+| 1.5          |
++--------------+
+
+select 10 % 3;
++--------+
+| 10 % 3 |
++--------+
+| 1      |
++--------+
+
+select 5.5 % 2;
++---------+
+| 5.5 % 2 |
++---------+
+| 1.5     |
++---------+
+
+select mod(9.9,3.3);
++---------------+
+| mod(9.9, 3.3) |
++---------------+
+| 0.0           |
++---------------+
+
+select mod(9.9,3);
++--------------------+
+| mod(9.9, 3)        |
++--------------------+
+| 0.8999996185302734 |
++--------------------+
+
+select mod(9.9, cast(3 as decimal(2,1)));
++-----------------------------------+
+| mod(9.9, cast(3 as decimal(2,1))) |
++-----------------------------------+
+| 0.9                               |
++-----------------------------------+
+
+select mod(9.9,3.0);
++---------------+
+| mod(9.9, 3.0) |
++---------------+
+| 0.9           |
++---------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.4.0" id="negative">
+
+        <dt rev="2.0.1">
+          <codeph>negative(numeric_type a)</codeph>
+<!-- <codeph>negative(int a), negative(double a), negative(decimal(p,s) a)</codeph> -->
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">negative() function</indexterm>
+          <b>Purpose:</b> Returns the argument with the sign reversed; returns a positive value if the argument was
+          already negative.
+          <p rev="2.0.1" conref="../shared/impala_common.xml#common/return_type_same"/>
+<!--
+            <p>
+              <b>Return type:</b> <codeph>int</codeph>, <codeph>double</codeph>,
+              or <codeph>decimal(p,s)</codeph> depending on type of argument
+            </p>
+            -->
+          <p>
+            <b>Usage notes:</b> Use <codeph>-abs(a)</codeph> instead if you need to ensure all return values are
+            negative.
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="pi">
+
+        <dt rev="1.4.0">
+          <codeph>pi()</codeph>
+        </dt>
+
+        <dd rev="1.4.0">
+          <indexterm audience="Cloudera">pi() function</indexterm>
+          <b>Purpose:</b> Returns the constant pi.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="pmod">
+
+        <dt>
+          <codeph>pmod(bigint a, bigint b), pmod(double a, double b)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">pmod() function</indexterm>
+          <b>Purpose:</b> Returns the positive modulus of a number.
+          Primarily for <xref href="https://issues.apache.org/jira/browse/HIVE-656" scope="external" format="html">HiveQL compatibility</xref>.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph> or <codeph>double</codeph>, depending on type of arguments
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples show how the <codeph>fmod()</codeph> function sometimes returns a negative value
+            depending on the sign of its arguments, and the <codeph>pmod()</codeph> function returns the same value
+            as <codeph>fmod()</codeph>, but sometimes with the sign flipped.
+          </p>
+<codeblock>select fmod(-5,2);
++-------------+
+| fmod(-5, 2) |
++-------------+
+| -1          |
++-------------+
+
+select pmod(-5,2);
++-------------+
+| pmod(-5, 2) |
++-------------+
+| 1           |
++-------------+
+
+select fmod(-5,-2);
++--------------+
+| fmod(-5, -2) |
++--------------+
+| -1           |
++--------------+
+
+select pmod(-5,-2);
++--------------+
+| pmod(-5, -2) |
++--------------+
+| -1           |
++--------------+
+
+select fmod(5,-2);
++-------------+
+| fmod(5, -2) |
++-------------+
+| 1           |
++-------------+
+
+select pmod(5,-2);
++-------------+
+| pmod(5, -2) |
++-------------+
+| -1          |
++-------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.4.0" id="positive">
+
+        <dt rev="2.0.1">
+          <codeph>positive(numeric_type a)</codeph>
+<!-- <codeph>positive(int a), positive(double a), positive(decimal(p,s) a</codeph> -->
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">positive() function</indexterm>
+          <b>Purpose:</b> Returns the original argument unchanged (even if the argument is negative).
+          <p rev="2.0.1" conref="../shared/impala_common.xml#common/return_type_same"/>
+<!--
+            <p>
+              <b>Return type:</b> <codeph>int</codeph>, <codeph>double</codeph>,
+              or <codeph>decimal(p,s)</codeph> depending on type of argument
+            </p>
+            -->
+          <p>
+            <b>Usage notes:</b> Use <codeph>abs()</codeph> instead if you need to ensure all return values are
+            positive.
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="pow">
+
+        <dt rev="1.4.0">
+          <codeph>pow(double a, double p)</codeph>,
+          <codeph id="power">power(double a, double p)</codeph>,
+          <codeph rev="2.3.0" id="dpow">dpow(double a, double p)</codeph>,
+          <codeph rev="2.3.0" id="fpow">fpow(double a, double p)</codeph>
+        </dt>
+
+        <dd rev="1.4.0">
+          <indexterm audience="Cloudera">pow() function</indexterm>
+          <indexterm audience="Cloudera">power() function</indexterm>
+          <indexterm audience="Cloudera">dpow() function</indexterm>
+          <indexterm audience="Cloudera">fpow() function</indexterm>
+          <b>Purpose:</b> Returns the first argument raised to the power of the second argument.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.4.0" id="precision">
+
+        <dt rev="1.4.0">
+          <codeph>precision(<varname>numeric_expression</varname>)</codeph>
+        </dt>
+
+        <dd rev="1.4.0">
+          <indexterm audience="Cloudera">precision() function</indexterm>
+          <b>Purpose:</b> Computes the precision (number of decimal digits) needed to represent the type of the
+          argument expression as a <codeph>DECIMAL</codeph> value.
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            Typically used in combination with the <codeph>scale()</codeph> function, to determine the appropriate
+            <codeph>DECIMAL(<varname>precision</varname>,<varname>scale</varname>)</codeph> type to declare in a
+            <codeph>CREATE TABLE</codeph> statement or <codeph>CAST()</codeph> function.
+          </p>
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p conref="../shared/impala_common.xml#common/precision_scale_example"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="quotient">
+
+        <dt>
+          <codeph>quotient(int numerator, int denominator)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">quotient() function</indexterm>
+          <b>Purpose:</b> Returns the first argument divided by the second argument, discarding any fractional
+          part. Avoids promoting arguments to <codeph>DOUBLE</codeph> as happens with the <codeph>/</codeph> SQL
+          operator.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="radians">
+
+        <dt>
+          <codeph>radians(double a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">radians() function</indexterm>
+          <b>Purpose:</b> Converts argument value from degrees to radians.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="rand">
+
+        <dt>
+          <codeph>rand()</codeph>, <codeph>rand(int seed)</codeph>,
+          <codeph rev="2.3.0" id="random">random()</codeph>,
+          <codeph rev="2.3.0">random(int seed)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">rand() function</indexterm>
+          <b>Purpose:</b> Returns a random value between 0 and 1. After <codeph>rand()</codeph> is called with a
+          seed argument, it produces a consistent random sequence based on the seed value.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+          <p>
+            <b>Usage notes:</b> Currently, the random sequence is reset after each query, and multiple calls to
+            <codeph>rand()</codeph> within the same query return the same value each time. For different number
+            sequences that are different for each query, pass a unique seed value to each call to
+            <codeph>rand()</codeph>. For example, <codeph>select rand(unix_timestamp()) from ...</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p>
+            The following examples show how <codeph>rand()</codeph> can produce sequences of varying predictability,
+            so that you can reproduce query results involving random values or generate unique sequences of random
+            values for each query.
+            When <codeph>rand()</codeph> is called with no argument, it generates the same sequence of values each time,
+            regardless of the ordering of the result set.
+            When <codeph>rand()</codeph> is called with a constant integer, it generates a different sequence of values,
+            but still always the same sequence for the same seed value.
+            If you pass in a seed value that changes, such as the return value of the expression <codeph>unix_timestamp(now())</codeph>,
+            each query will use a different sequence of random values, potentially more useful in probability calculations although
+            more difficult to reproduce at a later time. Therefore, the final two examples with an unpredictable seed value
+            also include the seed in the result set, to make it possible to reproduce the same random sequence later.
+          </p>
+<codeblock>select x, rand() from three_rows;
++---+-----------------------+
+| x | rand()                |
++---+-----------------------+
+| 1 | 0.0004714746030380365 |
+| 2 | 0.5895895192351144    |
+| 3 | 0.4431900859080209    |
++---+-----------------------+
+
+select x, rand() from three_rows order by x desc;
++---+-----------------------+
+| x | rand()                |
++---+-----------------------+
+| 3 | 0.0004714746030380365 |
+| 2 | 0.5895895192351144    |
+| 1 | 0.4431900859080209    |
++---+-----------------------+
+
+select x, rand(1234) from three_rows order by x;
++---+----------------------+
+| x | rand(1234)           |
++---+----------------------+
+| 1 | 0.7377511392057646   |
+| 2 | 0.009428468537250751 |
+| 3 | 0.208117277924026    |
++---+----------------------+
+
+select x, rand(1234) from three_rows order by x desc;
++---+----------------------+
+| x | rand(1234)           |
++---+----------------------+
+| 3 | 0.7377511392057646   |
+| 2 | 0.009428468537250751 |
+| 1 | 0.208117277924026    |
++---+----------------------+
+
+select x, unix_timestamp(now()), rand(unix_timestamp(now()))
+  from three_rows order by x;
++---+-----------------------+-----------------------------+
+| x | unix_timestamp(now()) | rand(unix_timestamp(now())) |
++---+-----------------------+-----------------------------+
+| 1 | 1440777752            | 0.002051228658320023        |
+| 2 | 1440777752            | 0.5098743483004506          |
+| 3 | 1440777752            | 0.9517714925817081          |
++---+-----------------------+-----------------------------+
+
+select x, unix_timestamp(now()), rand(unix_timestamp(now()))
+  from three_rows order by x desc;
++---+-----------------------+-----------------------------+
+| x | unix_timestamp(now()) | rand(unix_timestamp(now())) |
++---+-----------------------+-----------------------------+
+| 3 | 1440777761            | 0.9985985015512437          |
+| 2 | 1440777761            | 0.3251255333074953          |
+| 1 | 1440777761            | 0.02422675025846192         |
++---+-----------------------+-----------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="round">
+
+        <dt>
+          <codeph>round(double a)</codeph>,
+          <codeph>round(double a, int d)</codeph>,
+          <codeph rev="1.4.0">round(decimal a, int_type d)</codeph>,
+          <codeph rev="2.3.0" id="dround">dround(double a)</codeph>,
+          <codeph rev="2.3.0">dround(double a, int d)</codeph>,
+          <codeph rev="2.3.0">dround(decimal(p,s) a, int_type d)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">round() function</indexterm>
+          <indexterm audience="Cloudera">dround() function</indexterm>
+          <b>Purpose:</b> Rounds a floating-point value. By default (with a single argument), rounds to the nearest
+          integer. Values ending in .5 are rounded up for positive numbers, down for negative numbers (that is,
+          away from zero). The optional second argument specifies how many digits to leave after the decimal point;
+          values greater than zero produce a floating-point return value rounded to the requested number of digits
+          to the right of the decimal point.
+          <p rev="1.4.0">
+            <b>Return type:</b> <codeph>bigint</codeph> for single <codeph>double</codeph> argument.
+            <codeph>double</codeph> for two-argument signature when second argument greater than zero.
+            For <codeph>DECIMAL</codeph> values, the smallest
+            <codeph>DECIMAL(<varname>p</varname>,<varname>s</varname>)</codeph> type with appropriate precision and
+            scale.
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.4.0" id="scale">
+
+        <dt rev="1.4.0">
+          <codeph>scale(<varname>numeric_expression</varname>)</codeph>
+        </dt>
+
+        <dd rev="1.4.0">
+          <indexterm audience="Cloudera">scale() function</indexterm>
+          <b>Purpose:</b> Computes the scale (number of decimal digits to the right of the decimal point) needed to
+          represent the type of the argument expression as a <codeph>DECIMAL</codeph> value.
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            Typically used in combination with the <codeph>precision()</codeph> function, to determine the
+            appropriate <codeph>DECIMAL(<varname>precision</varname>,<varname>scale</varname>)</codeph> type to
+            declare in a <codeph>CREATE TABLE</codeph> statement or <codeph>CAST()</codeph> function.
+          </p>
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+          <p conref="../shared/impala_common.xml#common/precision_scale_example"/>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="sign">
+
+        <dt>
+          <codeph>sign(double a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">sign() function</indexterm>
+          <b>Purpose:</b> Returns -1, 0, or 1 to indicate the signedness of the argument value.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="sin">
+
+        <dt>
+          <codeph>sin(double a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">sin() function</indexterm>
+          <b>Purpose:</b> Returns the sine of the argument.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="sinh" rev="2.3.0 IMPALA-1771">
+
+        <dt rev="2.3.0 IMPALA-1771">
+          <codeph>sinh(double a)</codeph>
+        </dt>
+
+        <dd rev="2.3.0 IMPALA-1771">
+          <indexterm audience="Cloudera">sinh() function</indexterm>
+          <b>Purpose:</b> Returns the hyperbolic sine of the argument.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="sqrt">
+
+        <dt>
+          <codeph>sqrt(double a)</codeph>,
+          <codeph rev="2.3.0" id="dsqrt">dsqrt(double a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">sqrt() function</indexterm>
+          <indexterm audience="Cloudera">dsqrt() function</indexterm>
+          <b>Purpose:</b> Returns the square root of the argument.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="tan">
+
+        <dt>
+          <codeph>tan(double a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">tan() function</indexterm>
+          <b>Purpose:</b> Returns the tangent of the argument.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="tanh" rev="2.3.0 IMPALA-1771">
+
+        <dt rev="2.3.0 IMPALA-1771">
+          <codeph>tanh(double a)</codeph>
+        </dt>
+
+        <dd rev="2.3.0 IMPALA-1771">
+          <indexterm audience="Cloudera">tanh() function</indexterm>
+          <b>Purpose:</b> Returns the hyperbolic tangent of the argument.
+          <p>
+            <b>Return type:</b> <codeph>double</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="2.3.0" id="truncate">
+
+        <dt rev="2.3.0">
+          <codeph>truncate(double_or_decimal a[, digits_to_leave])</codeph>,
+          <ph id="dtrunc"><codeph>dtrunc(double_or_decimal a[, digits_to_leave])</codeph></ph>
+        </dt>
+
+        <dd rev="2.3.0">
+          <indexterm audience="Cloudera">truncate() function</indexterm>
+          <indexterm audience="Cloudera">dtrunc() function</indexterm>
+          <b>Purpose:</b> Removes some or all fractional digits from a numeric value.
+          With no argument, removes all fractional digits, leaving an integer value.
+          The optional argument specifies the number of fractional digits to include
+          in the return value, and only applies with the argument type is <codeph>DECIMAL</codeph>.
+          <codeph>truncate()</codeph> and <codeph>dtrunc()</codeph> are aliases for the same function.
+          <p>
+            <b>Return type:</b> <codeph>decimal</codeph> for <codeph>DECIMAL</codeph> arguments;
+            <codeph>bigint</codeph> for <codeph>DOUBLE</codeph> arguments
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>select truncate(3.45)
++----------------+
+| truncate(3.45) |
++----------------+
+| 3              |
++----------------+
+
+select truncate(-3.45)
++-----------------+
+| truncate(-3.45) |
++-----------------+
+| -3              |
++-----------------+
+
+select truncate(3.456,1)
++--------------------+
+| truncate(3.456, 1) |
++--------------------+
+| 3.4                |
++--------------------+
+
+select dtrunc(3.456,1)
++------------------+
+| dtrunc(3.456, 1) |
++------------------+
+| 3.4              |
++------------------+
+
+select truncate(3.456,2)
++--------------------+
+| truncate(3.456, 2) |
++--------------------+
+| 3.45               |
++--------------------+
+
+select truncate(3.456,7)
++--------------------+
+| truncate(3.456, 7) |
++--------------------+
+| 3.4560000          |
++--------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="unhex">
+
+        <dt>
+          <codeph>unhex(string a)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">unhex() function</indexterm>
+          <b>Purpose:</b> Returns a string of characters with ASCII values corresponding to pairs of hexadecimal
+          digits in the argument.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+    </dl>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_max.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_max.xml b/docs/topics/impala_max.xml
new file mode 100644
index 0000000..3f7b827
--- /dev/null
+++ b/docs/topics/impala_max.xml
@@ -0,0 +1,198 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="max">
+
+  <title>MAX Function</title>
+  <titlealts audience="PDF"><navtitle>MAX</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="Analytic Functions"/>
+      <data name="Category" value="Aggregate Functions"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">max() function</indexterm>
+      An aggregate function that returns the maximum value from a set of numbers. Opposite of the
+      <codeph>MIN</codeph> function. Its single argument can be numeric column, or the numeric result of a function
+      or expression applied to the column value. Rows with a <codeph>NULL</codeph> value for the specified column
+      are ignored. If the table is empty, or all the values supplied to <codeph>MAX</codeph> are
+      <codeph>NULL</codeph>, <codeph>MAX</codeph> returns <codeph>NULL</codeph>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>MAX([DISTINCT | ALL] <varname>expression</varname>) [OVER (<varname>analytic_clause</varname>)]</codeblock>
+
+    <p>
+      When the query contains a <codeph>GROUP BY</codeph> clause, returns one value for each combination of
+      grouping values.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_sliding_window"/>
+
+    <p conref="../shared/impala_common.xml#common/return_type_same_except_string"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/partition_key_optimization"/>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_aggregation_explanation"/>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_aggregation_example"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>-- Find the largest value for this column in the table.
+select max(c1) from t1;
+-- Find the largest value for this column from a subset of the table.
+select max(c1) from t1 where month = 'January' and year = '2013';
+-- Find the largest value from a set of numeric function results.
+select max(length(s)) from t1;
+-- Can also be used in combination with DISTINCT and/or GROUP BY.
+-- Return more than one result.
+select month, year, max(purchase_price) from store_stats group by month, year;
+-- Filter the input to eliminate duplicates before performing the calculation.
+select max(distinct x) from t1;
+</codeblock>
+
+    <p rev="2.0.0">
+      The following examples show how to use <codeph>MAX()</codeph> in an analytic context. They use a table
+      containing integers from 1 to 10. Notice how the <codeph>MAX()</codeph> is reported for each input value, as
+      opposed to the <codeph>GROUP BY</codeph> clause which condenses the result set.
+<codeblock>select x, property, max(x) over (partition by property) as max from int_t where property in ('odd','even');
++----+----------+-----+
+| x  | property | max |
++----+----------+-----+
+| 2  | even     | 10  |
+| 4  | even     | 10  |
+| 6  | even     | 10  |
+| 8  | even     | 10  |
+| 10 | even     | 10  |
+| 1  | odd      | 9   |
+| 3  | odd      | 9   |
+| 5  | odd      | 9   |
+| 7  | odd      | 9   |
+| 9  | odd      | 9   |
++----+----------+-----+
+</codeblock>
+
+Adding an <codeph>ORDER BY</codeph> clause lets you experiment with results that are cumulative or apply to a moving
+set of rows (the <q>window</q>). The following examples use <codeph>MAX()</codeph> in an analytic context
+(that is, with an <codeph>OVER()</codeph> clause) to display the smallest value of <codeph>X</codeph>
+encountered up to each row in the result set. The examples use two columns in the <codeph>ORDER BY</codeph>
+clause to produce a sequence of values that rises and falls, to illustrate how the <codeph>MAX()</codeph>
+result only increases or stays the same throughout each partition within the result set.
+The basic <codeph>ORDER BY x</codeph> clause implicitly
+activates a window clause of <codeph>RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW</codeph>,
+which is effectively the same as <codeph>ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW</codeph>,
+therefore all of these examples produce the same results:
+
+<codeblock>select x, property,
+  max(x) <b>over (order by property, x desc)</b> as 'maximum to this point'
+from int_t where property in ('prime','square');
++---+----------+-----------------------+
+| x | property | maximum to this point |
++---+----------+-----------------------+
+| 7 | prime    | 7                     |
+| 5 | prime    | 7                     |
+| 3 | prime    | 7                     |
+| 2 | prime    | 7                     |
+| 9 | square   | 9                     |
+| 4 | square   | 9                     |
+| 1 | square   | 9                     |
++---+----------+-----------------------+
+
+select x, property,
+  max(x) over
+  (
+    <b>order by property, x desc</b>
+    <b>rows between unbounded preceding and current row</b>
+  ) as 'maximum to this point'
+from int_t where property in ('prime','square');
++---+----------+-----------------------+
+| x | property | maximum to this point |
++---+----------+-----------------------+
+| 7 | prime    | 7                     |
+| 5 | prime    | 7                     |
+| 3 | prime    | 7                     |
+| 2 | prime    | 7                     |
+| 9 | square   | 9                     |
+| 4 | square   | 9                     |
+| 1 | square   | 9                     |
++---+----------+-----------------------+
+
+select x, property,
+  max(x) over
+  (
+    <b>order by property, x desc</b>
+    <b>range between unbounded preceding and current row</b>
+  ) as 'maximum to this point'
+from int_t where property in ('prime','square');
++---+----------+-----------------------+
+| x | property | maximum to this point |
++---+----------+-----------------------+
+| 7 | prime    | 7                     |
+| 5 | prime    | 7                     |
+| 3 | prime    | 7                     |
+| 2 | prime    | 7                     |
+| 9 | square   | 9                     |
+| 4 | square   | 9                     |
+| 1 | square   | 9                     |
++---+----------+-----------------------+
+</codeblock>
+
+The following examples show how to construct a moving window, with a running maximum taking into account all rows before
+and 1 row after the current row.
+Because of a restriction in the Impala <codeph>RANGE</codeph> syntax, this type of
+moving window is possible with the <codeph>ROWS BETWEEN</codeph> clause but not the <codeph>RANGE BETWEEN</codeph> clause.
+Because of an extra Impala restriction on the <codeph>MAX()</codeph> and <codeph>MIN()</codeph> functions in an
+analytic context, the lower bound must be <codeph>UNBOUNDED PRECEDING</codeph>.
+<codeblock>select x, property,
+  max(x) over
+  (
+    <b>order by property, x</b>
+    <b>rows between unbounded preceding and 1 following</b>
+  ) as 'local maximum'
+from int_t where property in ('prime','square');
++---+----------+---------------+
+| x | property | local maximum |
++---+----------+---------------+
+| 2 | prime    | 3             |
+| 3 | prime    | 5             |
+| 5 | prime    | 7             |
+| 7 | prime    | 7             |
+| 1 | square   | 7             |
+| 4 | square   | 9             |
+| 9 | square   | 9             |
++---+----------+---------------+
+
+-- Doesn't work because of syntax restriction on RANGE clause.
+select x, property,
+  max(x) over
+  (
+    <b>order by property, x</b>
+    <b>range between unbounded preceding and 1 following</b>
+  ) as 'local maximum'
+from int_t where property in ('prime','square');
+ERROR: AnalysisException: RANGE is only supported with both the lower and upper bounds UNBOUNDED or one UNBOUNDED and the other CURRENT ROW.
+</codeblock>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_analytic_functions.xml#analytic_functions"/>, <xref href="impala_min.xml#min"/>,
+      <xref href="impala_avg.xml#avg"/>
+    </p>
+  </conbody>
+</concept>



[04/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_sync_ddl.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_sync_ddl.xml b/docs/topics/impala_sync_ddl.xml
new file mode 100644
index 0000000..7453e94
--- /dev/null
+++ b/docs/topics/impala_sync_ddl.xml
@@ -0,0 +1,59 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.2.1" id="sync_ddl">
+
+  <title>SYNC_DDL Query Option</title>
+  <titlealts audience="PDF"><navtitle>SYNC_DDL</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">SYNC_DDL query option</indexterm>
+      When enabled, causes any DDL operation such as <codeph>CREATE TABLE</codeph> or <codeph>ALTER TABLE</codeph>
+      to return only when the changes have been propagated to all other Impala nodes in the cluster by the Impala
+      catalog service. That way, if you issue a subsequent <codeph>CONNECT</codeph> statement in
+      <cmdname>impala-shell</cmdname> to connect to a different node in the cluster, you can be sure that other
+      node will already recognize any added or changed tables. (The catalog service automatically broadcasts the
+      DDL changes to all nodes automatically, but without this option there could be a period of inconsistency if
+      you quickly switched to another node, such as by issuing a subsequent query through a load-balancing proxy.)
+    </p>
+
+    <p>
+      Although <codeph>INSERT</codeph> is classified as a DML statement, when the <codeph>SYNC_DDL</codeph> option
+      is enabled, <codeph>INSERT</codeph> statements also delay their completion until all the underlying data and
+      metadata changes are propagated to all Impala nodes. Internally, Impala inserts have similarities with DDL
+      statements in traditional database systems, because they create metadata needed to track HDFS block locations
+      for new files and they potentially add new partitions to partitioned tables.
+    </p>
+
+    <note>
+      Because this option can introduce a delay after each write operation, if you are running a sequence of
+      <codeph>CREATE DATABASE</codeph>, <codeph>CREATE TABLE</codeph>, <codeph>ALTER TABLE</codeph>,
+      <codeph>INSERT</codeph>, and similar statements within a setup script, to minimize the overall delay you can
+      enable the <codeph>SYNC_DDL</codeph> query option only near the end, before the final DDL statement.
+    </note>
+
+    <p conref="../shared/impala_common.xml#common/type_boolean"/>
+    <p conref="../shared/impala_common.xml#common/default_false_0"/>
+
+    <!-- To do:
+      Example could be useful here.
+    -->
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_ddl.xml#ddl"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_tables.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_tables.xml b/docs/topics/impala_tables.xml
new file mode 100644
index 0000000..80b4000
--- /dev/null
+++ b/docs/topics/impala_tables.xml
@@ -0,0 +1,258 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="tables">
+
+  <title>Overview of Impala Tables</title>
+  <titlealts audience="PDF"><navtitle>Tables</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Databases"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p/>
+
+    <p>
+      Tables are the primary containers for data in Impala. They have the familiar row and column layout similar to
+      other database systems, plus some features such as partitioning often associated with higher-end data
+      warehouse systems.
+    </p>
+
+    <p>
+      Logically, each table has a structure based on the definition of its columns, partitions, and other
+      properties.
+    </p>
+
+    <p>
+      Physically, each table that uses HDFS storage is associated with a directory in HDFS. The table data consists of all the data files
+      underneath that directory:
+    </p>
+
+    <ul>
+      <li>
+        <xref href="impala_tables.xml#internal_tables">Internal tables</xref> are managed by Impala, and use directories
+        inside the designated Impala work area.
+      </li>
+
+      <li>
+        <xref href="impala_tables.xml#external_tables">External tables</xref> use arbitrary HDFS directories, where
+        the data files are typically shared between different Hadoop components.
+      </li>
+
+      <li>
+        Large-scale data is usually handled by partitioned tables, where the data files are divided among different
+        HDFS subdirectories.
+      </li>
+    </ul>
+
+    <p rev="2.2.0">
+      Impala tables can also represent data that is stored in HBase, or in the Amazon S3 filesystem (CDH 5.4.0 or higher),
+      or on Isilon storage devices (CDH 5.4.3 or higher).  See <xref href="impala_hbase.xml#impala_hbase"/>,
+      <xref href="impala_s3.xml#s3"/>, and <xref href="impala_isilon.xml#impala_isilon"/>
+      for details about those special kinds of tables.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/ignore_file_extensions"/>
+
+    <p>
+      <b>Related statements:</b> <xref href="impala_create_table.xml#create_table"/>,
+      <xref href="impala_drop_table.xml#drop_table"/>, <xref href="impala_alter_table.xml#alter_table"/>
+      <xref href="impala_insert.xml#insert"/>, <xref href="impala_load_data.xml#load_data"/>,
+      <xref href="impala_select.xml#select"/>
+    </p>
+  </conbody>
+
+  <concept id="internal_tables">
+
+    <title>Internal Tables</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">internal tables</indexterm>
+        The default kind of table produced by the <codeph>CREATE TABLE</codeph> statement is known as an internal
+        table. (Its counterpart is the external table, produced by the <codeph>CREATE EXTERNAL TABLE</codeph>
+        syntax.)
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            Impala creates a directory in HDFS to hold the data files.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            You can create data in internal tables by issuing <codeph>INSERT</codeph> or <codeph>LOAD DATA</codeph>
+            statements.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            If you add or replace data using HDFS operations, issue the <codeph>REFRESH</codeph> command in
+            <cmdname>impala-shell</cmdname> so that Impala recognizes the changes in data files, block locations,
+            and so on.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            When you issue a <codeph>DROP TABLE</codeph> statement, Impala physically removes all the data files
+            from the directory.
+          </p>
+        </li>
+
+        <li>
+          <p conref="../shared/impala_common.xml#common/check_internal_external_table"/>
+        </li>
+
+        <li>
+          <p>
+            When you issue an <codeph>ALTER TABLE</codeph> statement to rename an internal table, all data files
+            are moved into the new HDFS directory for the table. The files are moved even if they were formerly in
+            a directory outside the Impala data directory, for example in an internal table with a
+            <codeph>LOCATION</codeph> attribute pointing to an outside HDFS directory.
+          </p>
+        </li>
+      </ul>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/switch_internal_external_table"/>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_tables.xml#external_tables"/>, <xref href="impala_create_table.xml#create_table"/>,
+        <xref href="impala_drop_table.xml#drop_table"/>, <xref href="impala_alter_table.xml#alter_table"/>,
+        <xref href="impala_describe.xml#describe"/>
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="external_tables">
+
+    <title>External Tables</title>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">external tables</indexterm>
+        The syntax <codeph>CREATE EXTERNAL TABLE</codeph> sets up an Impala table that points at existing data
+        files, potentially in HDFS locations outside the normal Impala data directories.. This operation saves the
+        expense of importing the data into a new table when you already have the data files in a known location in
+        HDFS, in the desired file format.
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            You can use Impala to query the data in this table.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            You can create data in external tables by issuing <codeph>INSERT</codeph> or <codeph>LOAD DATA</codeph>
+            statements.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            If you add or replace data using HDFS operations, issue the <codeph>REFRESH</codeph> command in
+            <cmdname>impala-shell</cmdname> so that Impala recognizes the changes in data files, block locations,
+            and so on.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            When you issue a <codeph>DROP TABLE</codeph> statement in Impala, that removes the connection that
+            Impala has with the associated data files, but does not physically remove the underlying data. You can
+            continue to use the data files with other Hadoop components and HDFS operations.
+          </p>
+        </li>
+
+        <li>
+          <p conref="../shared/impala_common.xml#common/check_internal_external_table"/>
+        </li>
+
+        <li>
+          <p>
+            When you issue an <codeph>ALTER TABLE</codeph> statement to rename an external table, all data files
+            are left in their original locations.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            You can point multiple external tables at the same HDFS directory by using the same
+            <codeph>LOCATION</codeph> attribute for each one. The tables could have different column definitions,
+            as long as the number and types of columns are compatible with the schema evolution considerations for
+            the underlying file type. For example, for text data files, one table might define a certain column as
+            a <codeph>STRING</codeph> while another defines the same column as a <codeph>BIGINT</codeph>.
+          </p>
+        </li>
+      </ul>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/switch_internal_external_table"/>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_tables.xml#internal_tables"/>, <xref href="impala_create_table.xml#create_table"/>,
+        <xref href="impala_drop_table.xml#drop_table"/>, <xref href="impala_alter_table.xml#alter_table"/>,
+        <xref href="impala_describe.xml#describe"/>
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="table_file_formats">
+    <title>File Formats</title>
+    <conbody>
+      <p>
+        Each table has an associated file format, which determines how Impala interprets the
+        associated data files. See <xref href="impala_file_formats.xml#file_formats"/> for details.
+      </p>
+      <p>
+        You set the file format during the <codeph>CREATE TABLE</codeph> statement,
+        or change it later using the <codeph>ALTER TABLE</codeph> statement.
+        Partitioned tables can have a different file format for individual partitions,
+        allowing you to change the file format used in your ETL process for new data
+        without going back and reconverting all the existing data in the same table.
+      </p>
+      <p>
+        Any <codeph>INSERT</codeph> statements produce new data files with the current file format of the table.
+        For existing data files, changing the file format of the table does not automatically do any data conversion.
+        You must use <codeph>TRUNCATE TABLE</codeph> or <codeph>INSERT OVERWRITE</codeph> to remove any previous data
+        files that use the old file format.
+        Then you use the <codeph>LOAD DATA</codeph> statement, <codeph>INSERT ... SELECT</codeph>, or other mechanism
+        to put data files of the correct format into the table.
+      </p>
+      <p>
+        The default file format, text, is the most flexible and easy to produce when you are just getting started with
+        Impala. The Parquet file format offers the highest query performance and uses compression to reduce storage
+        requirements; therefore, <ph rev="upstream">Cloudera</ph> recommends using Parquet for Impala tables with substantial amounts of data.
+        <ph rev="2.3.0">Also, the complex types (<codeph>ARRAY</codeph>, <codeph>STRUCT</codeph>, and <codeph>MAP</codeph>)
+        available in <keyword keyref="impala23_full"/> and higher are currently only supported with the Parquet file type.</ph>
+        Based on your existing ETL workflow, you might use other file formats such as Avro, possibly doing a final
+        conversion step to Parquet to take advantage of its performance for analytic queries.
+      </p>
+    </conbody>
+  </concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_timeouts.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_timeouts.xml b/docs/topics/impala_timeouts.xml
new file mode 100644
index 0000000..03ad773
--- /dev/null
+++ b/docs/topics/impala_timeouts.xml
@@ -0,0 +1,172 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.2.1" id="timeouts">
+
+  <title>Setting Timeout Periods for Daemons, Queries, and Sessions</title>
+
+  <titlealts audience="PDF">
+
+    <navtitle>Setting Timeouts</navtitle>
+
+  </titlealts>
+
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Scheduling"/>
+      <data name="Category" value="Scalability"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Depending on how busy your CDH cluster is, you might increase or decrease various timeout
+      values. Increase timeouts if Impala is cancelling operations prematurely, when the system
+      is responding slower than usual but the operations are still successful if given extra
+      time. Decrease timeouts if operations are idle or hanging for long periods, and the idle
+      or hung operations are consuming resources and reducing concurrency.
+    </p>
+
+    <p outputclass="toc inpage"/>
+
+  </conbody>
+
+  <concept id="statestore_timeout">
+
+    <title>Increasing the Statestore Timeout</title>
+
+    <conbody>
+
+      <p rev="IMP-1210">
+        If you have an extensive Impala schema, for example with hundreds of databases, tens of
+        thousands of tables, and so on, you might encounter timeout errors during startup as the
+        Impala catalog service broadcasts metadata to all the Impala nodes using the statestore
+        service. To avoid such timeout errors on startup, increase the statestore timeout value
+        from its default of 10 seconds. Specify the timeout value using the
+        <codeph>-statestore_subscriber_timeout_seconds</codeph> option for the statestore
+        service, using the configuration instructions in
+        <xref href="impala_config_options.xml#config_options"/>. The symptom of this problem is
+        messages in the <codeph>impalad</codeph> log such as:
+      </p>
+
+<codeblock>Connection with state-store lost
+Trying to re-register with state-store</codeblock>
+
+      <p>
+        See <xref href="impala_scalability.xml#statestore_scalability"/> for more details about
+        statestore operation and settings on clusters with a large number of Impala-related
+        objects such as tables and partitions.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="impalad_timeout">
+
+    <title>Setting the Idle Query and Idle Session Timeouts for impalad</title>
+
+    <conbody>
+
+      <p>
+        To keep long-running queries or idle sessions from tying up cluster resources, you can
+        set timeout intervals for both individual queries, and entire sessions.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/timeout_clock_blurb"/>
+
+      <p>
+        Specify the following startup options for the <cmdname>impalad</cmdname> daemon:
+      </p>
+
+      <ul>
+        <li>
+          The <codeph>--idle_query_timeout</codeph> option specifies the time in seconds after
+          which an idle query is cancelled. This could be a query whose results were all fetched
+          but was never closed, or one whose results were partially fetched and then the client
+          program stopped requesting further results. This condition is most likely to occur in
+          a client program using the JDBC or ODBC interfaces, rather than in the interactive
+          <cmdname>impala-shell</cmdname> interpreter. Once the query is cancelled, the client
+          program cannot retrieve any further results.
+        </li>
+
+        <li>
+          The <codeph>--idle_session_timeout</codeph> option specifies the time in seconds after
+          which an idle session is expired. A session is idle when no activity is occurring for
+          any of the queries in that session, and the session has not started any new queries.
+          Once a session is expired, you cannot issue any new query requests to it. The session
+          remains open, but the only operation you can perform is to close it. The default value
+          of 0 means that sessions never expire.
+        </li>
+      </ul>
+
+      <p>
+        For instructions on changing <cmdname>impalad</cmdname> startup options, see
+        <xref href="impala_config_options.xml#config_options"/>.
+      </p>
+
+      <p rev="2.0.0">
+        You can reduce the idle query timeout by using the <codeph>QUERY_TIMEOUT_S</codeph>
+        query option. Any value specified for the <codeph>--idle_query_timeout</codeph> startup
+        option serves as an upper limit for the <codeph>QUERY_TIMEOUT_S</codeph> query option.
+        See <xref href="impala_query_timeout_s.xml#query_timeout_s"/> for details.
+      </p>
+
+    </conbody>
+
+  </concept>
+  <concept id="concept_rfy_jl1_rx">
+    <title>Setting Timeout and Retries for Thrift Connections to the Backend
+      Client</title>
+    <conbody>
+      <p>Impala connections to the backend client are subject to failure in
+        cases when the network is momentarily overloaded. To avoid failed
+        queries due to transient network problems, you can configure the number
+        of Thrift connection retries using the following option: </p>
+      <ul id="ul_bj3_ql1_rx">
+        <li>The <codeph>--backend_client_connection_num_retries</codeph> option
+          specifies the number of times Impala will try connecting to the
+          backend client after the first connection attempt fails. By default,
+            <cmdname>impalad</cmdname> will attempt three re-connections before
+          it returns a failure. </li>
+      </ul>
+      <p>You can configure timeouts for sending and receiving data from the
+        backend client. Therefore, if for some reason a query hangs, instead of
+        waiting indefinitely for a response, Impala will terminate the
+        connection after a configurable timeout.</p>
+      <ul id="ul_vm2_2v1_rx">
+        <li>The <codeph>--backend_client_rpc_timeout_ms</codeph> option can be
+          used to specify the number of milliseconds Impala should wait for a
+          response from the backend client before it terminates the connection
+          and signals a failure. The default value for this property is 300000
+          milliseconds, or 5 minutes. </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="cancel_query">
+
+    <title>Cancelling a Query</title>
+
+    <conbody>
+
+      <p>
+        Sometimes, an Impala query might run for an unexpectedly long time, tying up resources
+        in the cluster. You can cancel the query explicitly, independent of the timeout period,
+        by going into the web UI for the <cmdname>impalad</cmdname> host (on port 25000 by
+        default), and using the link on the <codeph>/queries</codeph> tab to cancel the running
+        query. Various client applications let you interactively cancel queries submitted or
+        monitored through those applications. For example, by pressing <codeph>^C</codeph> in
+        <cmdname>impala-shell</cmdname>, clicking the <uicontrol>Cancel</uicontrol> button from
+        the <uicontrol>Watch</uicontrol> page in Hue, clicking <uicontrol>Actions &gt;
+        Cancel</uicontrol> from the <uicontrol>Queries</uicontrol> list in Cloudera Manager, and
+        so on.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_timestamp.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_timestamp.xml b/docs/topics/impala_timestamp.xml
new file mode 100644
index 0000000..2f4fa40
--- /dev/null
+++ b/docs/topics/impala_timestamp.xml
@@ -0,0 +1,445 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="timestamp">
+
+  <title>TIMESTAMP Data Type</title>
+  <titlealts audience="PDF"><navtitle>TIMESTAMP</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Dates and Times"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      A data type used in <codeph>CREATE TABLE</codeph> and <codeph>ALTER TABLE</codeph> statements, representing a
+      point in time.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p>
+      In the column definition of a <codeph>CREATE TABLE</codeph> statement:
+    </p>
+
+<codeblock><varname>column_name</varname> TIMESTAMP</codeblock>
+
+    <p>
+      <b>Range:</b> Allowed date values range from 1400-01-01 to 9999-12-31; this range is different from the Hive
+      <codeph>TIMESTAMP</codeph> type. Internally, the resolution of the time portion of a
+      <codeph>TIMESTAMP</codeph> value is in nanoseconds.
+    </p>
+
+    <p>
+      <b>INTERVAL expressions:</b>
+    </p>
+
+    <p>
+      You can perform date arithmetic by adding or subtracting a specified number of time units, using the
+      <codeph>INTERVAL</codeph> keyword and the <codeph>+</codeph> and <codeph>-</codeph> operators or
+      <codeph>date_add()</codeph> and <codeph>date_sub()</codeph> functions. You can specify units as
+      <codeph>YEAR[S]</codeph>, <codeph>MONTH[S]</codeph>, <codeph>WEEK[S]</codeph>, <codeph>DAY[S]</codeph>,
+      <codeph>HOUR[S]</codeph>, <codeph>MINUTE[S]</codeph>, <codeph>SECOND[S]</codeph>,
+      <codeph>MILLISECOND[S]</codeph>, <codeph>MICROSECOND[S]</codeph>, and <codeph>NANOSECOND[S]</codeph>. You can
+      only specify one time unit in each interval expression, for example <codeph>INTERVAL 3 DAYS</codeph> or
+      <codeph>INTERVAL 25 HOURS</codeph>, but you can produce any granularity by adding together successive
+      <codeph>INTERVAL</codeph> values, such as <codeph><varname>timestamp_value</varname> + INTERVAL 3 WEEKS -
+      INTERVAL 1 DAY + INTERVAL 10 MICROSECONDS</codeph>.
+    </p>
+
+    <p>
+      For example:
+    </p>
+
+<codeblock>select now() + interval 1 day;
+select date_sub(now(), interval 5 minutes);
+insert into auction_details
+  select auction_id, auction_start_time, auction_start_time + interval 2 days + interval 12 hours
+  from new_auctions;</codeblock>
+
+    <p>
+      <b>Time zones:</b>
+    </p>
+
+    <p>
+      By default, Impala does not store timestamps using the local timezone, to avoid undesired results from
+      unexpected time zone issues. Timestamps are stored and interpreted relative to UTC, both when written to or
+      read from data files, or when converted to or from Unix time values through functions such as
+      <codeph>from_unixtime()</codeph> or <codeph>unix_timestamp()</codeph>. To convert such a
+      <codeph>TIMESTAMP</codeph> value to one that represents the date and time in a specific time zone, convert
+      the original value with the <codeph>from_utc_timestamp()</codeph> function.
+    </p>
+
+    <p>
+      Because Impala does not assume that <codeph>TIMESTAMP</codeph> values are in any particular time zone, you
+      must be conscious of the time zone aspects of data that you query, insert, or convert.
+    </p>
+
+    <p>
+      For consistency with Unix system calls, the <codeph>TIMESTAMP</codeph> returned by the <codeph>now()</codeph>
+      function represents the local time in the system time zone, rather than in UTC. To store values relative to
+      the current time in a portable way, convert any <codeph>now()</codeph> return values using the
+      <codeph>to_utc_timestamp()</codeph> function first. For example, the following example shows that the current
+      time in California (where this Impala cluster is located) is shortly after 2 PM. If that value was written to a data
+      file, and shipped off to a distant server to be analyzed alongside other data from far-flung locations, the
+      dates and times would not match up precisely because of time zone differences. Therefore, the
+      <codeph>to_utc_timestamp()</codeph> function converts it using a common reference point, the UTC time zone
+      (descended from the old Greenwich Mean Time standard). The <codeph>'PDT'</codeph> argument indicates that the
+      original value is from the Pacific time zone with Daylight Saving Time in effect. When servers in all
+      geographic locations run the same transformation on any local date and time values (with the appropriate time
+      zone argument), the stored data uses a consistent representation. Impala queries can use functions such as
+      <codeph>EXTRACT()</codeph>, <codeph>MIN()</codeph>, <codeph>AVG()</codeph>, and so on to do time-series
+      analysis on those timestamps.
+    </p>
+
+<codeblock>[localhost:21000] > select now();
++-------------------------------+
+| now()                         |
++-------------------------------+
+| 2015-04-09 14:07:46.580465000 |
++-------------------------------+
+[localhost:21000] > select to_utc_timestamp(now(), 'PDT');
++--------------------------------+
+| to_utc_timestamp(now(), 'pdt') |
++--------------------------------+
+| 2015-04-09 21:08:07.664547000  |
++--------------------------------+
+</codeblock>
+
+    <p>
+      The converse function, <codeph>from_utc_timestamp()</codeph>, lets you take stored <codeph>TIMESTAMP</codeph>
+      data or calculated results and convert back to local date and time for processing on the application side.
+      The following example shows how you might represent some future date (such as the ending date and time of an
+      auction) in UTC, and then convert back to local time when convenient for reporting or other processing. The
+      final query in the example tests whether this arbitrary UTC date and time has passed yet, by converting it
+      back to the local time zone and comparing it against the current date and time.
+    </p>
+
+<codeblock>[localhost:21000] > select to_utc_timestamp(now() + interval 2 weeks, 'PDT');
++---------------------------------------------------+
+| to_utc_timestamp(now() + interval 2 weeks, 'pdt') |
++---------------------------------------------------+
+| 2015-04-23 21:08:34.152923000                     |
++---------------------------------------------------+
+[localhost:21000] > select from_utc_timestamp('2015-04-23 21:08:34.152923000','PDT');
++------------------------------------------------------------+
+| from_utc_timestamp('2015-04-23 21:08:34.152923000', 'pdt') |
++------------------------------------------------------------+
+| 2015-04-23 14:08:34.152923000                              |
++------------------------------------------------------------+
+[localhost:21000] > select from_utc_timestamp('2015-04-23 21:08:34.152923000','PDT') &lt; now();
++--------------------------------------------------------------------+
+| from_utc_timestamp('2015-04-23 21:08:34.152923000', 'pdt') &lt; now() |
++--------------------------------------------------------------------+
+| false                                                              |
++--------------------------------------------------------------------+
+</codeblock>
+
+    <p rev="2.2.0">
+      If you have data files written by Hive, those <codeph>TIMESTAMP</codeph> values represent the local timezone
+      of the host where the data was written, potentially leading to inconsistent results when processed by Impala.
+      To avoid compatibility problems or having to code workarounds, you can specify one or both of these
+      <cmdname>impalad</cmdname> startup flags: <codeph>-use_local_tz_for_unix_timestamp_conversions=true</codeph>
+      <codeph>-convert_legacy_hive_parquet_utc_timestamps=true</codeph>. Although
+      <codeph>-convert_legacy_hive_parquet_utc_timestamps</codeph> is turned off by default to avoid performance overhead, <ph rev="upstream">Cloudera</ph> recommends
+      turning it on when processing <codeph>TIMESTAMP</codeph> columns in Parquet files written by Hive, to avoid unexpected behavior.
+    </p>
+
+    <p rev="2.2.0">
+      The <codeph>-use_local_tz_for_unix_timestamp_conversions</codeph> setting affects conversions from
+      <codeph>TIMESTAMP</codeph> to <codeph>BIGINT</codeph>, or from <codeph>BIGINT</codeph>
+      to <codeph>TIMESTAMP</codeph>. By default, Impala treats all <codeph>TIMESTAMP</codeph> values as UTC,
+      to simplify analysis of time-series data from different geographic regions. When you enable the
+      <codeph>-use_local_tz_for_unix_timestamp_conversions</codeph> setting, these operations
+      treat the input values as if they are in the local tie zone of the host doing the processing.
+      See <xref href="impala_datetime_functions.xml#datetime_functions"/> for the list of functions
+      affected by the <codeph>-use_local_tz_for_unix_timestamp_conversions</codeph> setting.
+    </p>
+
+    <p>
+      The following sequence of examples shows how the interpretation of <codeph>TIMESTAMP</codeph> values in
+      Parquet tables is affected by the setting of the <codeph>-convert_legacy_hive_parquet_utc_timestamps</codeph>
+      setting.
+    </p>
+
+    <p>
+      Regardless of the <codeph>-convert_legacy_hive_parquet_utc_timestamps</codeph> setting,
+      <codeph>TIMESTAMP</codeph> columns in text tables can be written and read interchangeably by Impala and Hive:
+    </p>
+
+<codeblock>Impala DDL and queries for text table:
+
+[localhost:21000] > create table t1 (x timestamp);
+[localhost:21000] > insert into t1 values (now()), (now() + interval 1 day);
+[localhost:21000] > select x from t1;
++-------------------------------+
+| x                             |
++-------------------------------+
+| 2015-04-07 15:43:02.892403000 |
+| 2015-04-08 15:43:02.892403000 |
++-------------------------------+
+[localhost:21000] > select to_utc_timestamp(x, 'PDT') from t1;
++-------------------------------+
+| to_utc_timestamp(x, 'pdt')    |
++-------------------------------+
+| 2015-04-07 22:43:02.892403000 |
+| 2015-04-08 22:43:02.892403000 |
++-------------------------------+
+
+Hive query for text table:
+
+hive> select * from t1;
+OK
+2015-04-07 15:43:02.892403
+2015-04-08 15:43:02.892403
+Time taken: 1.245 seconds, Fetched: 2 row(s)
+</codeblock>
+
+    <p>
+      When the table uses Parquet format, Impala expects any time zone adjustment to be applied prior to writing,
+      while <codeph>TIMESTAMP</codeph> values written by Hive are adjusted to be in the UTC time zone. When Hive
+      queries Parquet data files that it wrote, it adjusts the <codeph>TIMESTAMP</codeph> values back to the local
+      time zone, while Impala does no conversion. Hive does no time zone conversion when it queries Impala-written
+      Parquet files.
+    </p>
+
+<codeblock>Impala DDL and queries for Parquet table:
+
+[localhost:21000] > create table p1 stored as parquet as select x from t1;
++-------------------+
+| summary           |
++-------------------+
+| Inserted 2 row(s) |
++-------------------+
+[localhost:21000] > select x from p1;
++-------------------------------+
+| x                             |
++-------------------------------+
+| 2015-04-07 15:43:02.892403000 |
+| 2015-04-08 15:43:02.892403000 |
++-------------------------------+
+
+Hive DDL and queries for Parquet table:
+
+hive> create table h1 (x timestamp) stored as parquet;
+OK
+hive> insert into h1 select * from p1;
+...
+OK
+Time taken: 35.573 seconds
+hive> select x from p1;
+OK
+2015-04-07 15:43:02.892403
+2015-04-08 15:43:02.892403
+Time taken: 0.324 seconds, Fetched: 2 row(s)
+hive> select x from h1;
+OK
+2015-04-07 15:43:02.892403
+2015-04-08 15:43:02.892403
+Time taken: 0.197 seconds, Fetched: 2 row(s)
+</codeblock>
+
+    <p>
+      The discrepancy arises when Impala queries the Hive-created Parquet table. The underlying values in the
+      <codeph>TIMESTAMP</codeph> column are different from the ones written by Impala, even though they were copied
+      from one table to another by an <codeph>INSERT ... SELECT</codeph> statement in Hive. Hive did an implicit
+      conversion from the local time zone to UTC as it wrote the values to Parquet.
+    </p>
+
+<codeblock>Impala query for TIMESTAMP values from Impala-written and Hive-written data:
+
+[localhost:21000] > select * from p1;
++-------------------------------+
+| x                             |
++-------------------------------+
+| 2015-04-07 15:43:02.892403000 |
+| 2015-04-08 15:43:02.892403000 |
++-------------------------------+
+Fetched 2 row(s) in 0.29s
+[localhost:21000] > select * from h1;
++-------------------------------+
+| x                             |
++-------------------------------+
+| 2015-04-07 22:43:02.892403000 |
+| 2015-04-08 22:43:02.892403000 |
++-------------------------------+
+Fetched 2 row(s) in 0.41s
+
+Underlying integer values for Impala-written and Hive-written data:
+
+[localhost:21000] > select cast(x as bigint) from p1;
++-------------------+
+| cast(x as bigint) |
++-------------------+
+| 1428421382        |
+| 1428507782        |
++-------------------+
+Fetched 2 row(s) in 0.38s
+[localhost:21000] > select cast(x as bigint) from h1;
++-------------------+
+| cast(x as bigint) |
++-------------------+
+| 1428446582        |
+| 1428532982        |
++-------------------+
+Fetched 2 row(s) in 0.20s
+</codeblock>
+
+    <p>
+      When the <codeph>-convert_legacy_hive_parquet_utc_timestamps</codeph> setting is enabled, Impala recognizes
+      the Parquet data files written by Hive, and applies the same UTC-to-local-timezone conversion logic during
+      the query as Hive uses, making the contents of the Impala-written <codeph>P1</codeph> table and the
+      Hive-written <codeph>H1</codeph> table appear identical, whether represented as <codeph>TIMESTAMP</codeph>
+      values or the underlying <codeph>BIGINT</codeph> integers:
+    </p>
+
+<codeblock>[localhost:21000] > select x from p1;
++-------------------------------+
+| x                             |
++-------------------------------+
+| 2015-04-07 15:43:02.892403000 |
+| 2015-04-08 15:43:02.892403000 |
++-------------------------------+
+Fetched 2 row(s) in 0.37s
+[localhost:21000] > select x from h1;
++-------------------------------+
+| x                             |
++-------------------------------+
+| 2015-04-07 15:43:02.892403000 |
+| 2015-04-08 15:43:02.892403000 |
++-------------------------------+
+Fetched 2 row(s) in 0.19s
+[localhost:21000] > select cast(x as bigint) from p1;
++-------------------+
+| cast(x as bigint) |
++-------------------+
+| 1428446582        |
+| 1428532982        |
++-------------------+
+Fetched 2 row(s) in 0.29s
+[localhost:21000] > select cast(x as bigint) from h1;
++-------------------+
+| cast(x as bigint) |
++-------------------+
+| 1428446582        |
+| 1428532982        |
++-------------------+
+Fetched 2 row(s) in 0.22s
+</codeblock>
+
+    <p>
+      <b>Conversions:</b>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/timestamp_conversions"/>
+
+    <p>
+      In Impala 1.3 and higher, the <codeph>FROM_UNIXTIME()</codeph> and <codeph>UNIX_TIMESTAMP()</codeph>
+      functions allow a wider range of format strings, with more flexibility in element order, repetition of letter
+      placeholders, and separator characters. In <keyword keyref="impala23_full"/> and higher, the <codeph>UNIX_TIMESTAMP()</codeph>
+      function also allows a numeric timezone offset to be specified as part of the input string.
+      See <xref href="impala_datetime_functions.xml#datetime_functions"/> for details.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/y2k38"/>
+
+    <p>
+      <b>Partitioning:</b>
+    </p>
+
+    <p>
+      Although you cannot use a <codeph>TIMESTAMP</codeph> column as a partition key, you can extract the
+      individual years, months, days, hours, and so on and partition based on those columns. Because the partition
+      key column values are represented in HDFS directory names, rather than as fields in the data files
+      themselves, you can also keep the original <codeph>TIMESTAMP</codeph> values if desired, without duplicating
+      data or wasting storage space. See <xref href="impala_partitioning.xml#partition_key_columns"/> for more
+      details on partitioning with date and time values.
+    </p>
+
+<codeblock>[localhost:21000] &gt; create table timeline (event string) partitioned by (happened timestamp);
+ERROR: AnalysisException: Type 'TIMESTAMP' is not supported as partition-column type in column: happened
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>select cast('1966-07-30' as timestamp);
+select cast('1985-09-25 17:45:30.005' as timestamp);
+select cast('08:30:00' as timestamp);
+select hour('1970-01-01 15:30:00');         -- Succeeds, returns 15.
+select hour('1970-01-01 15:30');            -- Returns NULL because seconds field required.
+select hour('1970-01-01 27:30:00');         -- Returns NULL because hour value out of range.
+select dayofweek('2004-06-13');             -- Returns 1, representing Sunday.
+select dayname('2004-06-13');               -- Returns 'Sunday'.
+select date_add('2004-06-13', 365);         -- Returns 2005-06-13 with zeros for hh:mm:ss fields.
+select day('2004-06-13');                   -- Returns 13.
+select datediff('1989-12-31','1984-09-01'); -- How many days between these 2 dates?
+select now();                               -- Returns current date and time in local timezone.
+
+create table dates_and_times (t timestamp);
+insert into dates_and_times values
+  ('1966-07-30'), ('1985-09-25 17:45:30.005'), ('08:30:00'), (now());
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/null_bad_timestamp_cast"/>
+
+    <p conref="../shared/impala_common.xml#common/partitioning_worrisome"/>
+
+    <p conref="../shared/impala_common.xml#common/hbase_ok"/>
+
+    <p conref="../shared/impala_common.xml#common/parquet_ok"/>
+
+    <p conref="../shared/impala_common.xml#common/text_bulky"/>
+
+<!--    <p conref="../shared/impala_common.xml#common/compatibility_blurb"/> -->
+
+    <p conref="../shared/impala_common.xml#common/internals_16_bytes"/>
+
+    <p conref="../shared/impala_common.xml#common/added_forever"/>
+
+    <p conref="../shared/impala_common.xml#common/column_stats_constant"/>
+
+    <p conref="../shared/impala_common.xml#common/sqoop_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/sqoop_timestamp_caveat"/>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+    <p>
+      If you cast a <codeph>STRING</codeph> with an unrecognized format to a <codeph>TIMESTAMP</codeph>, the result
+      is <codeph>NULL</codeph> rather than an error. Make sure to test your data pipeline to be sure any textual
+      date and time values are in a format that Impala <codeph>TIMESTAMP</codeph> can recognize.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/avro_no_timestamp"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <ul>
+      <li>
+<!-- The Timestamp Literals topic is pretty brief. Consider adding more examples there. -->
+        <xref href="impala_literals.xml#timestamp_literals"/>.
+      </li>
+
+      <li>
+        To convert to or from different date formats, or perform date arithmetic, use the date and time functions
+        described in <xref href="impala_datetime_functions.xml#datetime_functions"/>. In particular, the
+        <codeph>from_unixtime()</codeph> function requires a case-sensitive format string such as
+        <codeph>"yyyy-MM-dd HH:mm:ss.SSSS"</codeph>, matching one of the allowed variations of a
+        <codeph>TIMESTAMP</codeph> value (date plus time, only date, only time, optional fractional seconds).
+      </li>
+
+      <li>
+        See <xref href="impala_langref_unsupported.xml#langref_hiveql_delta"/> for details about differences in
+        <codeph>TIMESTAMP</codeph> handling between Impala and Hive.
+      </li>
+    </ul>
+
+  </conbody>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_tinyint.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_tinyint.xml b/docs/topics/impala_tinyint.xml
new file mode 100644
index 0000000..2756cb8
--- /dev/null
+++ b/docs/topics/impala_tinyint.xml
@@ -0,0 +1,103 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="tinyint">
+
+  <title>TINYINT Data Type</title>
+  <titlealts audience="PDF"><navtitle>TINYINT</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      A 1-byte integer data type used in <codeph>CREATE TABLE</codeph> and <codeph>ALTER TABLE</codeph> statements.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p>
+      In the column definition of a <codeph>CREATE TABLE</codeph> statement:
+    </p>
+
+<codeblock><varname>column_name</varname> TINYINT</codeblock>
+
+    <p>
+      <b>Range:</b> -128 .. 127. There is no <codeph>UNSIGNED</codeph> subtype.
+    </p>
+
+    <p>
+      <b>Conversions:</b> Impala automatically converts to a larger integer type (<codeph>SMALLINT</codeph>,
+      <codeph>INT</codeph>, or <codeph>BIGINT</codeph>) or a floating-point type (<codeph>FLOAT</codeph> or
+      <codeph>DOUBLE</codeph>) automatically. Use <codeph>CAST()</codeph> to convert to <codeph>STRING</codeph> or
+      <codeph>TIMESTAMP</codeph>.
+      <ph conref="../shared/impala_common.xml#common/cast_int_to_timestamp"/>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/int_overflow_behavior"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      For a convenient and automated way to check the bounds of the <codeph>TINYINT</codeph> type, call the
+      functions <codeph>MIN_TINYINT()</codeph> and <codeph>MAX_TINYINT()</codeph>.
+    </p>
+
+    <p>
+      If an integer value is too large to be represented as a <codeph>TINYINT</codeph>, use a
+      <codeph>SMALLINT</codeph> instead.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/null_bad_numeric_cast"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>CREATE TABLE t1 (x TINYINT);
+SELECT CAST(100 AS TINYINT);
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/parquet_blurb"/>
+
+<!-- Duplicated under TINYINT and SMALLINT. Turn into a conref in both places. -->
+
+    <p rev="1.4.0">
+      Physically, Parquet files represent <codeph>TINYINT</codeph> and <codeph>SMALLINT</codeph> values as 32-bit
+      integers. Although Impala rejects attempts to insert out-of-range values into such columns, if you create a
+      new table with the <codeph>CREATE TABLE ... LIKE PARQUET</codeph> syntax, any <codeph>TINYINT</codeph> or
+      <codeph>SMALLINT</codeph> columns in the original table turn into <codeph>INT</codeph> columns in the new
+      table.
+    </p>
+
+<!-- <p conref="../shared/impala_common.xml#common/partitioning_blurb"/> -->
+
+    <p conref="../shared/impala_common.xml#common/hbase_ok"/>
+
+    <p conref="../shared/impala_common.xml#common/text_bulky"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/compatibility_blurb"/> -->
+
+    <p conref="../shared/impala_common.xml#common/internals_1_bytes"/>
+
+    <p conref="../shared/impala_common.xml#common/added_forever"/>
+
+    <p conref="../shared/impala_common.xml#common/column_stats_constant"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/restrictions_blurb"/> -->
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_literals.xml#numeric_literals"/>, <xref href="impala_tinyint.xml#tinyint"/>,
+      <xref href="impala_smallint.xml#smallint"/>, <xref href="impala_int.xml#int"/>,
+      <xref href="impala_bigint.xml#bigint"/>, <xref href="impala_decimal.xml#decimal"/>,
+      <xref href="impala_math_functions.xml#math_functions"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_trouble_bad_results.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_trouble_bad_results.xml b/docs/topics/impala_trouble_bad_results.xml
new file mode 100644
index 0000000..dc15c5f
--- /dev/null
+++ b/docs/topics/impala_trouble_bad_results.xml
@@ -0,0 +1,25 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="trouble_bad_results">
+
+  <title>Troubleshooting Incorrect Results</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">incorrect results</indexterm>
+      <indexterm audience="Cloudera">bad results</indexterm>
+      <indexterm audience="Cloudera">wrong results</indexterm>
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_trouble_memory.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_trouble_memory.xml b/docs/topics/impala_trouble_memory.xml
new file mode 100644
index 0000000..ae6ab2a
--- /dev/null
+++ b/docs/topics/impala_trouble_memory.xml
@@ -0,0 +1,25 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="trouble_memory">
+
+  <title>Troubleshooting Out-of-Memory Issues</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Memory"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">memory errors</indexterm>
+      <indexterm audience="Cloudera">out-of-memory errors</indexterm>
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_trouble_query_fail.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_trouble_query_fail.xml b/docs/topics/impala_trouble_query_fail.xml
new file mode 100644
index 0000000..6b1d1ee
--- /dev/null
+++ b/docs/topics/impala_trouble_query_fail.xml
@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="trouble_query_fail">
+
+  <title>Troubleshooting Query Errors</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">query errors</indexterm>
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_trouble_sql.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_trouble_sql.xml b/docs/topics/impala_trouble_sql.xml
new file mode 100644
index 0000000..25d79f9
--- /dev/null
+++ b/docs/topics/impala_trouble_sql.xml
@@ -0,0 +1,25 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="trouble_sql">
+
+  <title>Troubleshooting SQL Syntax Errors</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">SQL errors</indexterm>
+      <indexterm audience="Cloudera">syntax errors</indexterm>
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_trouble_startup.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_trouble_startup.xml b/docs/topics/impala_trouble_startup.xml
new file mode 100644
index 0000000..ff4b49e
--- /dev/null
+++ b/docs/topics/impala_trouble_startup.xml
@@ -0,0 +1,25 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="trouble_startup">
+
+  <title>Troubleshooting Startup Problems</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Starting and Stopping"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">startup errors</indexterm>
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_troubleshooting.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_troubleshooting.xml b/docs/topics/impala_troubleshooting.xml
new file mode 100644
index 0000000..f7ebe4e
--- /dev/null
+++ b/docs/topics/impala_troubleshooting.xml
@@ -0,0 +1,447 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="troubleshooting">
+
+  <title>Troubleshooting Impala</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">troubleshooting</indexterm>
+      Troubleshooting for Impala requires being able to diagnose and debug problems
+      with performance, network connectivity, out-of-memory conditions, disk space usage,
+      and crash or hang conditions in any of the Impala-related daemons.
+    </p>
+
+    <p outputclass="toc inpage" audience="PDF">
+      The following sections describe the general troubleshooting procedures to diagnose
+      different kinds of problems:
+    </p>
+
+  </conbody>
+
+  <concept id="trouble_sql">
+
+    <title>Troubleshooting Impala SQL Syntax Issues</title>
+
+    <conbody>
+
+      <p>
+        In general, if queries issued against Impala fail, you can try running these same queries against Hive.
+      </p>
+
+      <ul>
+        <li>
+          If a query fails against both Impala and Hive, it is likely that there is a problem with your query or
+          other elements of your CDH environment:
+          <ul>
+            <li>
+              Review the <xref href="impala_langref.xml#langref">Language Reference</xref> to ensure your query is
+              valid.
+            </li>
+
+            <li>
+              Check <xref href="impala_reserved_words.xml#reserved_words"/> to see if any database, table,
+              column, or other object names in your query conflict with Impala reserved words.
+              Quote those names with backticks (<codeph>``</codeph>) if so.
+            </li>
+
+            <li>
+              Check <xref href="impala_functions.xml#builtins"/> to confirm whether Impala supports all the
+              built-in functions being used by your query, and whether argument and return types are the
+              same as you expect.
+            </li>
+
+            <li>
+              Review the <xref href="impala_logging.xml#logs_debug">contents of the Impala logs</xref> for any information that may be useful in identifying the
+              source of the problem.
+            </li>
+          </ul>
+        </li>
+
+        <li>
+          If a query fails against Impala but not Hive, it is likely that there is a problem with your Impala
+          installation.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="trouble_io" rev="CDH-19201">
+    <title>Troubleshooting I/O Capacity Problems</title>
+    <conbody>
+      <p>
+        Impala queries are typically I/O-intensive. If there is an I/O problem with storage devices,
+        or with HDFS itself, Impala queries could show slow response times with no obvious cause
+        on the Impala side. Slow I/O on even a single DataNode could result in an overall slowdown, because
+        queries involving clauses such as <codeph>ORDER BY</codeph>, <codeph>GROUP BY</codeph>, or <codeph>JOIN</codeph>
+        do not start returning results until all DataNodes have finished their work.
+      </p>
+      <p>
+        To test whether the Linux I/O system itself is performing as expected, run Linux commands like
+        the following on each DataNode:
+      </p>
+<codeblock>
+$ sudo sysctl -w vm.drop_caches=3 vm.drop_caches=0
+vm.drop_caches = 3
+vm.drop_caches = 0
+$ sudo dd if=/dev/sda bs=1M of=/dev/null count=1k 
+1024+0 records in
+1024+0 records out
+1073741824 bytes (1.1 GB) copied, 5.60373 s, 192 MB/s
+$ sudo dd if=/dev/sdb bs=1M of=/dev/null count=1k
+1024+0 records in
+1024+0 records out
+1073741824 bytes (1.1 GB) copied, 5.51145 s, 195 MB/s
+$ sudo dd if=/dev/sdc bs=1M of=/dev/null count=1k
+1024+0 records in
+1024+0 records out
+1073741824 bytes (1.1 GB) copied, 5.58096 s, 192 MB/s
+$ sudo dd if=/dev/sdd bs=1M of=/dev/null count=1k
+1024+0 records in
+1024+0 records out
+1073741824 bytes (1.1 GB) copied, 5.43924 s, 197 MB/s
+</codeblock>
+      <p>
+        On modern hardware, a throughput rate of less than 100 MB/s typically indicates
+        a performance issue with the storage device. Correct the hardware problem before
+        continuing with Impala tuning or benchmarking.
+      </p>
+    </conbody>
+  </concept>
+
+
+  <concept id="trouble_cookbook">
+
+    <title>Impala Troubleshooting Quick Reference</title>
+
+    <conbody>
+
+      <p>
+        The following table lists common problems and potential solutions.
+      </p>
+
+      <table>
+        <tgroup cols="3">
+          <colspec colname="1" colwidth="10*"/>
+          <colspec colname="2" colwidth="30*"/>
+          <colspec colname="3" colwidth="30*"/>
+          <thead>
+            <row>
+              <entry>
+                Symptom
+              </entry>
+              <entry>
+                Explanation
+              </entry>
+              <entry>
+                Recommendation
+              </entry>
+            </row>
+          </thead>
+          <tbody>
+            <row>
+              <entry>
+                Impala takes a long time to start.
+              </entry>
+              <entry>
+                Impala instances with large numbers of tables, partitions, or data files take longer to start
+                because the metadata for these objects is broadcast to all <cmdname>impalad</cmdname> nodes and
+                cached.
+              </entry>
+              <entry>
+                Adjust timeout and synchronicity settings.
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  Joins fail to complete.
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  There may be insufficient memory. During a join, data from the second, third, and so on sets to
+                  be joined is loaded into memory. If Impala chooses an inefficient join order or join mechanism,
+                  the query could exceed the total memory available.
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Start by gathering statistics with the <codeph>COMPUTE STATS</codeph> statement for each table
+                  involved in the join. Consider specifying the <codeph>[SHUFFLE]</codeph> hint so that data from
+                  the joined tables is split up between nodes rather than broadcast to each node. If tuning at the
+                  SQL level is not sufficient, add more memory to your system or join smaller data sets.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  Queries return incorrect results.
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Impala metadata may be outdated after changes are performed in Hive.
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Where possible, use the appropriate Impala statement (<codeph>INSERT</codeph>, <codeph>LOAD
+                  DATA</codeph>, <codeph>CREATE TABLE</codeph>, <codeph>ALTER TABLE</codeph>, <codeph>COMPUTE
+                  STATS</codeph>, and so on) rather than switching back and forth between Impala and Hive. Impala
+                  automatically broadcasts the results of DDL and DML operations to all Impala nodes in the
+                  cluster, but does not automatically recognize when such changes are made through Hive. After
+                  inserting data, adding a partition, or other operation in Hive, refresh the metadata for the
+                  table as described in <xref href="impala_refresh.xml#refresh"/>.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  Queries are slow to return results.
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Some <codeph>impalad</codeph> instances may not have started. Using a browser, connect to the
+                  host running the Impala state store. Connect using an address of the form
+                  <codeph>http://<varname>hostname</varname>:<varname>port</varname>/metrics</codeph>.
+                </p>
+
+                <p>
+                  <note>
+                    Replace <varname>hostname</varname> and <varname>port</varname> with the hostname and port of
+                    your Impala state store host machine and web server port. The default port is 25010.
+                  </note>
+                  The number of <codeph>impalad</codeph> instances listed should match the expected number of
+                  <codeph>impalad</codeph> instances installed in the cluster. There should also be one
+                  <codeph>impalad</codeph> instance installed on each DataNode
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Ensure Impala is installed on all DataNodes. Start any <codeph>impalad</codeph> instances that
+                  are not running.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  Queries are slow to return results.
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Impala may not be configured to use native checksumming. Native checksumming uses
+                  machine-specific instructions to compute checksums over HDFS data very quickly. Review Impala
+                  logs. If you find instances of "<codeph>INFO util.NativeCodeLoader: Loaded the
+                  native-hadoop</codeph>" messages, native checksumming is not enabled.
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Ensure Impala is configured to use native checksumming as described in
+                  <xref href="impala_config_performance.xml#config_performance"/>.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  Queries are slow to return results.
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Impala may not be configured to use data locality tracking.
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Test Impala for data locality tracking and make configuration changes as necessary. Information
+                  on this process can be found in <xref href="impala_config_performance.xml#config_performance"/>.
+                </p>
+              </entry>
+            </row>
+            <row>
+              <entry>
+                <p>
+                  Attempts to complete Impala tasks such as executing INSERT-SELECT actions fail. The Impala logs
+                  include notes that files could not be opened due to permission denied.
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  This can be the result of permissions issues. For example, you could use the Hive shell as the
+                  hive user to create a table. After creating this table, you could attempt to complete some
+                  action, such as an INSERT-SELECT on the table. Because the table was created using one user and
+                  the INSERT-SELECT is attempted by another, this action may fail due to permissions issues.
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  In general, ensure the Impala user has sufficient permissions. In the preceding example, ensure
+                  the Impala user has sufficient permissions to the table that the Hive user created.
+                </p>
+              </entry>
+            </row>
+            <row rev="IMP-1210">
+              <entry>
+                <p>
+                  Impala fails to start up, with the <cmdname>impalad</cmdname> logs referring to errors connecting
+                  to the statestore service and attempts to re-register.
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  A large number of databases, tables, partitions, and so on can require metadata synchronization,
+                  particularly on startup, that takes longer than the default timeout for the statestore service.
+                </p>
+              </entry>
+              <entry>
+                <p>
+                  Configure the statestore timeout value and possibly other settings related to the frequency of
+                  statestore updates and metadata loading. See
+                  <xref href="impala_timeouts.xml#statestore_timeout"/> and
+                  <xref href="impala_scalability.xml#statestore_scalability"/>.
+                </p>
+              </entry>
+            </row>
+          </tbody>
+        </tgroup>
+      </table>
+
+      <p audience="Cloudera">
+        Some or all of these settings might also be useful.
+<codeblock>NUM_SCANNER_THREADS: 0
+ABORT_ON_DEFAULT_LIMIT_EXCEEDED: 0
+MAX_IO_BUFFERS: 0
+DEFAULT_ORDER_BY_LIMIT: -1
+BATCH_SIZE: 0
+NUM_NODES: 0
+DISABLE_CODEGEN: 0
+MAX_ERRORS: 0
+ABORT_ON_ERROR: 0
+MAX_SCAN_RANGE_LENGTH: 0
+ALLOW_UNSUPPORTED_FORMATS: 0
+SUPPORT_START_OVER: false
+DEBUG_ACTION:
+MEM_LIMIT: 0
+</codeblock>
+      </p>
+    </conbody>
+  </concept>
+
+  <concept audience="Cloudera" id="core_dumps">
+
+    <title>Enabling Core Dumps for Impala</title>
+
+    <conbody>
+
+        <p>
+          Fill in details, then unhide.
+        </p>
+
+        <p>
+          From Nong:
+        </p>
+
+        <p>
+          In a CM-managed cluster, search for "core" from the impala configuration page. You should see the "enable
+          core dump" config.
+        </p>
+
+        <p>
+          From <xref href="impala_config_options.xml#config_options"/>:
+        </p>
+
+<codeblock>export ENABLE_CORE_DUMPS=${ENABLE_COREDUMPS:-false}</codeblock>
+
+        <note conref="../shared/impala_common.xml#common/core_dump_considerations"/>
+
+      <p></p>
+    </conbody>
+  </concept>
+
+  <concept audience="Cloudera" id="io_throughput">
+    <title>Verifying I/O Throughput</title>
+    <conbody>
+      <p>
+        Optimal Impala query performance depends on being able to perform I/O across multiple storage devices
+        in parallel, with the data transferred at or close to the maximum throughput for each device.
+        If a hardware or configuration issue causes a reduction in I/O throughput, even if the problem only
+        affects a subset of storage devices, you might experience
+        slow query performance that cannot be improved by using regular SQL tuning techniques.
+      </p>
+      <p>
+        As a general guideline, expect each commodity storage device (for example, a standard rotational
+        hard drive) to be able to transfer approximately 100 MB per second. If you see persistent slow query
+        perormance, examine the Impala logs to check
+      </p>
+
+<codeblock>
+<![CDATA[
+Useful test from beta at Visa.
+SME: Jayant@
+
+Symptoms:
+* Queries running slow
+* Scan rate of IO in Impala logs show noticeably less than expected IO rate for each disk (typical commodity disk should provide ~100 MB/s
+
+Actions:
+* Validate disk read from OS to confirm no issue at hardware or OS level
+* Validate disk read at HDFS to see if issue at HDFS config
+
+Specifics:
+Testing Linux and hardware IO:
+# First running:
+sudo sysctl -w vm.drop_caches=3 vm.drop_caches=0
+
+# Then Running:
+sudo dd if=/dev/sda bs=1M of=/dev/null count=1k
+& sudo dd if=/dev/sdb bs=1M of=/dev/null count=1k
+& sudo dd if=/dev/sdc bs=1M of=/dev/null count=1k
+& sudo dd if=/dev/sdd bs=1M of=/dev/null count=1k & wait
+
+Testing HDFS IO:
+# You can use TestDFSIO. Its documented here ; http://answers.oreilly.com/topic/460-how-to-benchmark-a-hadoop-cluster/
+# You can also use sar, dd and iostat for monitoring the disk.
+
+# writes 10 files each of 1000 MB
+hadoop jar $HADOOP_INSTALL/hadoop-*-test.jar TestDFSIO -write -nrFiles 10 -fileSize 1000
+
+# run the read benchmark
+hadoop jar $HADOOP_INSTALL/hadoop-*-test.jar TestDFSIO -read -nrFiles 10 -fileSize 1000
+
+# clean up the data
+hadoop jar $HADOOP_INSTALL/hadoop-*-test.jar TestDFSIO -clean
+]]>
+</codeblock>
+
+    </conbody>
+  </concept>
+
+  <concept id="webui_snippet" audience="PDF">
+    <title conref="impala_webui.xml#webui/webui_title"/>
+    <conbody>
+      <p conref="impala_webui.xml#webui/webui_intro"/>
+      <p>
+        For full details, see <xref href="impala_webui.xml#webui"/>.
+      </p>
+    </conbody>
+  </concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_truncate_table.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_truncate_table.xml b/docs/topics/impala_truncate_table.xml
new file mode 100644
index 0000000..72f072f
--- /dev/null
+++ b/docs/topics/impala_truncate_table.xml
@@ -0,0 +1,184 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="2.3.0" id="truncate_table">
+
+  <title>TRUNCATE TABLE Statement (<keyword keyref="impala23"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>TRUNCATE TABLE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="S3"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Disk Storage"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.3.0">
+      <indexterm audience="Cloudera">TRUNCATE TABLE statement</indexterm>
+      Removes the data from an Impala table while leaving the table itself.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>TRUNCATE TABLE <ph rev="2.5.0 IMPALA-2641">[IF EXISTS]</ph> [<varname>db_name</varname>.]<varname>table_name</varname></codeblock>
+
+    <p conref="../shared/impala_common.xml#common/ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      Often used to empty tables that are used during ETL cycles, after the data has been copied to another
+      table for the next stage of processing. This statement is a low-overhead alternative to dropping and
+      recreating the table, or using <codeph>INSERT OVERWRITE</codeph> to replace the data during the
+      next ETL cycle.
+    </p>
+
+    <p>
+      This statement removes all the data and associated data files in the table. It can remove data files from internal tables,
+      external tables, partitioned tables, and tables mapped to HBase or the Amazon Simple Storage Service (S3).
+      The data removal applies to the entire table, including all partitions of a partitioned table.
+    </p>
+
+    <p>
+      Any statistics produced by the <codeph>COMPUTE STATS</codeph> statement are reset when the data is removed.
+    </p>
+
+    <p>
+      Make sure that you are in the correct database before truncating a table, either by issuing a
+      <codeph>USE</codeph> statement first or by using a fully qualified name
+      <codeph><varname>db_name</varname>.<varname>table_name</varname></codeph>.
+    </p>
+
+    <p rev="2.5.0 IMPALA-2641">
+      The optional <codeph>IF EXISTS</codeph> clause makes the statement succeed whether or not the table exists.
+      If the table does exist, it is truncated; if it does not exist, the statement has no effect. This capability is
+      useful in standardized setup scripts that are might be run both before and after some of the tables exist.
+      This clause is available in <keyword keyref="impala25_full"/> and higher.
+    </p>
+
+    <p>
+      Any HDFS data files removed by this statement go into the HDFS trashcan, from which you can recover them
+      within a defined time interval if this operation turns out to be a mistake.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/disk_space_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/s3_blurb"/>
+    <p rev="2.2.0">
+      Although Impala cannot write new data to a table stored in the Amazon
+      S3 filesystem, the <codeph>TRUNCATE TABLE</codeph> statement can remove data files from S3.
+      See <xref href="impala_s3.xml#s3"/> for details about working with S3 tables.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+    <p rev="CDH-19187">
+      The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+      typically the <codeph>impala</codeph> user, must have write
+      permission for all the files and directories that make up the table.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      The following example shows a table containing some data and with table and column statistics.
+      After the <codeph>TRUNCATE TABLE</codeph> statement, the data is removed and the statistics
+      are reset.
+    </p>
+
+<codeblock>CREATE TABLE truncate_demo (x INT);
+INSERT INTO truncate_demo VALUES (1), (2), (4), (8);
+SELECT COUNT(*) FROM truncate_demo;
++----------+
+| count(*) |
++----------+
+| 4        |
++----------+
+COMPUTE STATS truncate_demo;
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 1 partition(s) and 1 column(s). |
++-----------------------------------------+
+SHOW TABLE STATS truncate_demo;
++-------+--------+------+--------------+-------------------+--------+-------------------+
+| #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats |
++-------+--------+------+--------------+-------------------+--------+-------------------+
+| 4     | 1      | 8B   | NOT CACHED   | NOT CACHED        | TEXT   | false             |
++-------+--------+------+--------------+-------------------+--------+-------------------+
+SHOW COLUMN STATS truncate_demo;
++--------+------+------------------+--------+----------+----------+
+| Column | Type | #Distinct Values | #Nulls | Max Size | Avg Size |
++--------+------+------------------+--------+----------+----------+
+| x      | INT  | 4                | -1     | 4        | 4        |
++--------+------+------------------+--------+----------+----------+
+
+-- After this statement, the data and the table/column stats will be gone.
+TRUNCATE TABLE truncate_demo;
+
+SELECT COUNT(*) FROM truncate_demo;
++----------+
+| count(*) |
++----------+
+| 0        |
++----------+
+SHOW TABLE STATS truncate_demo;
++-------+--------+------+--------------+-------------------+--------+-------------------+
+| #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats |
++-------+--------+------+--------------+-------------------+--------+-------------------+
+| -1    | 0      | 0B   | NOT CACHED   | NOT CACHED        | TEXT   | false             |
++-------+--------+------+--------------+-------------------+--------+-------------------+
+SHOW COLUMN STATS truncate_demo;
++--------+------+------------------+--------+----------+----------+
+| Column | Type | #Distinct Values | #Nulls | Max Size | Avg Size |
++--------+------+------------------+--------+----------+----------+
+| x      | INT  | -1               | -1     | 4        | 4        |
++--------+------+------------------+--------+----------+----------+
+</codeblock>
+
+    <p>
+      The following example shows how the <codeph>IF EXISTS</codeph> clause allows the <codeph>TRUNCATE TABLE</codeph>
+      statement to be run without error whether or not the table exists:
+    </p>
+
+<codeblock rev="2.5.0 IMPALA-2641">CREATE TABLE staging_table1 (x INT, s STRING);
+Fetched 0 row(s) in 0.33s
+
+SHOW TABLES LIKE 'staging*';
++----------------+
+| name           |
++----------------+
+| staging_table1 |
++----------------+
+Fetched 1 row(s) in 0.25s
+
+-- Our ETL process involves removing all data from several staging tables
+-- even though some might be already dropped, or not created yet.
+
+TRUNCATE TABLE IF EXISTS staging_table1;
+Fetched 0 row(s) in 5.04s
+
+TRUNCATE TABLE IF EXISTS staging_table2;
+Fetched 0 row(s) in 0.25s
+
+TRUNCATE TABLE IF EXISTS staging_table3;
+Fetched 0 row(s) in 0.25s
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_tables.xml#tables"/>,
+      <xref href="impala_alter_table.xml#alter_table"/>, <xref href="impala_create_table.xml#create_table"/>,
+      <xref href="impala_partitioning.xml#partitioning"/>, <xref href="impala_tables.xml#internal_tables"/>,
+      <xref href="impala_tables.xml#external_tables"/>
+    </p>
+
+  </conbody>
+</concept>


[33/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_drop_function.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_drop_function.xml b/docs/topics/impala_drop_function.xml
new file mode 100644
index 0000000..dfad51b
--- /dev/null
+++ b/docs/topics/impala_drop_function.xml
@@ -0,0 +1,127 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.2" id="drop_function">
+
+  <title>DROP FUNCTION Statement</title>
+  <titlealts audience="PDF"><navtitle>DROP FUNCTION</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="UDFs"/>
+      <data name="Category" value="Schemas"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">DROP FUNCTION statement</indexterm>
+      Removes a user-defined function (UDF), so that it is not available for execution during Impala
+      <codeph>SELECT</codeph> or <codeph>INSERT</codeph> operations.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p>
+      To drop C++ UDFs and UDAs:
+    </p>
+
+<codeblock>DROP [AGGREGATE] FUNCTION [IF EXISTS] [<varname>db_name</varname>.]<varname>function_name</varname>(<varname>type</varname>[, <varname>type</varname>...])</codeblock>
+
+    <note rev="2.5.0 IMPALA-2843 CDH-39148">
+      <p rev="2.5.0 IMPALA-2843 CDH-39148">
+        The preceding syntax, which includes the function signature, also applies to Java UDFs that were created
+        using the corresponding <codeph>CREATE FUNCTION</codeph> syntax that includes the argument and return types.
+        After upgrading to <keyword keyref="impala25_full"/> or higher, consider re-creating all Java UDFs with the
+        <codeph>CREATE FUNCTION</codeph> syntax that does not include the function signature. Java UDFs created this
+        way are now persisted in the metastore database and do not need to be re-created after an Impala restart.
+      </p>
+    </note>
+
+    <p rev="2.5.0 IMPALA-2843 CDH-39148">
+      To drop Java UDFs (created using the <codeph>CREATE FUNCTION</codeph> syntax with no function signature):
+    </p>
+
+<codeblock rev="2.5.0">DROP FUNCTION [IF EXISTS] [<varname>db_name</varname>.]<varname>function_name</varname></codeblock>
+
+<!--
+Examples:
+CREATE FUNCTION IF NOT EXISTS foo location '/path/to/jar' SYMBOL='TestUdf';
+CREATE FUNCTION bar location '/path/to/jar' SYMBOL='TestUdf2';
+DROP FUNCTION foo;
+DROP FUNCTION IF EXISTS bar;
+-->
+
+    <p conref="../shared/impala_common.xml#common/ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      Because the same function name could be overloaded with different argument signatures, you specify the
+      argument types to identify the exact function to drop.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/udf_persistence_restriction"/>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+    <p rev="CDH-19187">
+      The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+      typically the <codeph>impala</codeph> user, does not need any
+      particular HDFS permissions to perform this statement.
+      All read and write operations are on the metastore database,
+      not HDFS files and directories.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+    <p rev="2.5.0 IMPALA-2843 CDH-39148">
+      The following example shows how to drop Java functions created with the signatureless
+      <codeph>CREATE FUNCTION</codeph> syntax in <keyword keyref="impala25_full"/> and higher.
+      Issuing <codeph>DROP FUNCTION <varname>function_name</varname></codeph> removes all the
+      overloaded functions under that name.
+      (See <xref href="impala_create_function.xml#create_function"/> for a longer example
+      showing how to set up such functions in the first place.)
+    </p>
+<codeblock rev="2.5.0 IMPALA-2843 CDH-39148">
+create function my_func location '/user/impala/udfs/udf-examples-cdh570.jar'
+  symbol='com.cloudera.impala.TestUdf';
+
+show functions;
++-------------+---------------------------------------+-------------+---------------+
+| return type | signature                             | binary type | is persistent |
++-------------+---------------------------------------+-------------+---------------+
+| BIGINT      | my_func(BIGINT)                       | JAVA        | true          |
+| BOOLEAN     | my_func(BOOLEAN)                      | JAVA        | true          |
+| BOOLEAN     | my_func(BOOLEAN, BOOLEAN)             | JAVA        | true          |
+...
+| BIGINT      | testudf(BIGINT)                       | JAVA        | true          |
+| BOOLEAN     | testudf(BOOLEAN)                      | JAVA        | true          |
+| BOOLEAN     | testudf(BOOLEAN, BOOLEAN)             | JAVA        | true          |
+...
+
+drop function my_func;
+show functions;
++-------------+---------------------------------------+-------------+---------------+
+| return type | signature                             | binary type | is persistent |
++-------------+---------------------------------------+-------------+---------------+
+| BIGINT      | testudf(BIGINT)                       | JAVA        | true          |
+| BOOLEAN     | testudf(BOOLEAN)                      | JAVA        | true          |
+| BOOLEAN     | testudf(BOOLEAN, BOOLEAN)             | JAVA        | true          |
+...
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_udf.xml#udfs"/>, <xref href="impala_create_function.xml#create_function"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_drop_role.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_drop_role.xml b/docs/topics/impala_drop_role.xml
new file mode 100644
index 0000000..7383b46
--- /dev/null
+++ b/docs/topics/impala_drop_role.xml
@@ -0,0 +1,71 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.4.0" id="drop_role">
+
+  <title>DROP ROLE Statement (<keyword keyref="impala20"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>DROP ROLE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Sentry"/>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="Roles"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+      <!-- Consider whether to go deeper into categories like Security for the Sentry-related statements. -->
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">DROP ROLE statement</indexterm>
+<!-- Copied from Sentry docs. Turn into conref. I did some rewording for clarity. -->
+      The <codeph>DROP ROLE</codeph> statement removes a role from the metastore database. Once dropped, the role
+      is revoked for all users to whom it was previously assigned, and all privileges granted to that role are
+      revoked. Queries that are already executing are not affected. Impala verifies the role information
+      approximately every 60 seconds, so the effects of <codeph>DROP ROLE</codeph> might not take effect for new
+      Impala queries for a brief period.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>DROP ROLE <varname>role_name</varname>
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/privileges_blurb"/>
+
+    <p>
+      Only administrative users (initially, a predefined set of users specified in the Sentry service configuration
+      file) can use this statement.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/compatibility_blurb"/>
+
+    <p>
+      Impala makes use of any roles and privileges specified by the <codeph>GRANT</codeph> and
+      <codeph>REVOKE</codeph> statements in Hive, and Hive makes use of any roles and privileges specified by the
+      <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements in Impala. The Impala <codeph>GRANT</codeph>
+      and <codeph>REVOKE</codeph> statements for privileges do not require the <codeph>ROLE</codeph> keyword to be
+      repeated before each role name, unlike the equivalent Hive statements.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_authorization.xml#authorization"/>, <xref href="impala_grant.xml#grant"/>
+      <xref href="impala_revoke.xml#revoke"/>, <xref href="impala_create_role.xml#create_role"/>,
+      <xref href="impala_show.xml#show"/>
+    </p>
+
+<!-- To do: nail down the new SHOW syntax, e.g. SHOW ROLES, SHOW CURRENT ROLES, SHOW GROUPS. -->
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb_no"/>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_drop_stats.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_drop_stats.xml b/docs/topics/impala_drop_stats.xml
new file mode 100644
index 0000000..df82d6b
--- /dev/null
+++ b/docs/topics/impala_drop_stats.xml
@@ -0,0 +1,279 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="2.1.0" id="drop_stats">
+
+  <title>DROP STATS Statement</title>
+  <titlealts audience="PDF"><navtitle>DROP STATS</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="ETL"/>
+      <data name="Category" value="Ingest"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Scalability"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.1.0">
+      <indexterm audience="Cloudera">DROP STATS statement</indexterm>
+      Removes the specified statistics from a table or partition. The statistics were originally created by the
+      <codeph>COMPUTE STATS</codeph> or <codeph>COMPUTE INCREMENTAL STATS</codeph> statement.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock rev="2.1.0">DROP STATS [<varname>database_name</varname>.]<varname>table_name</varname>
+DROP INCREMENTAL STATS [<varname>database_name</varname>.]<varname>table_name</varname> PARTITION (<varname>partition_spec</varname>)
+
+<varname>partition_spec</varname> ::= <varname>partition_col</varname>=<varname>constant_value</varname>
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/incremental_partition_spec"/>
+
+    <p>
+      <codeph>DROP STATS</codeph> removes all statistics from the table, whether created by <codeph>COMPUTE
+      STATS</codeph> or <codeph>COMPUTE INCREMENTAL STATS</codeph>.
+    </p>
+
+    <p rev="2.1.0">
+      <codeph>DROP INCREMENTAL STATS</codeph> only affects incremental statistics for a single partition, specified
+      through the <codeph>PARTITION</codeph> clause. The incremental stats are marked as outdated, so that they are
+      recomputed by the next <codeph>COMPUTE INCREMENTAL STATS</codeph> statement.
+    </p>
+
+<!-- To do: what release was this added in? -->
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      You typically use this statement when the statistics for a table or a partition have become stale due to data
+      files being added to or removed from the associated HDFS data directories, whether by manual HDFS operations
+      or <codeph>INSERT</codeph>, <codeph>INSERT OVERWRITE</codeph>, or <codeph>LOAD DATA</codeph> statements, or
+      adding or dropping partitions.
+    </p>
+
+    <p>
+      When a table or partition has no associated statistics, Impala treats it as essentially zero-sized when
+      constructing the execution plan for a query. In particular, the statistics influence the order in which
+      tables are joined in a join query. To ensure proper query planning and good query performance and
+      scalability, make sure to run <codeph>COMPUTE STATS</codeph> or <codeph>COMPUTE INCREMENTAL STATS</codeph> on
+      the table or partition after removing any stale statistics.
+    </p>
+
+    <p>
+      Dropping the statistics is not required for an unpartitioned table or a partitioned table covered by the
+      original type of statistics. A subsequent <codeph>COMPUTE STATS</codeph> statement replaces any existing
+      statistics with new ones, for all partitions, regardless of whether the old ones were outdated. Therefore,
+      this statement was rarely used before the introduction of incremental statistics.
+    </p>
+
+    <p>
+      Dropping the statistics is required for a partitioned table containing incremental statistics, to make a
+      subsequent <codeph>COMPUTE INCREMENTAL STATS</codeph> statement rescan an existing partition. See
+      <xref href="impala_perf_stats.xml#perf_stats"/> for information about incremental statistics, a new feature
+      available in Impala 2.1.0 and higher.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+    <p rev="CDH-19187">
+      The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+      typically the <codeph>impala</codeph> user, does not need any
+      particular HDFS permissions to perform this statement.
+      All read and write operations are on the metastore database,
+      not HDFS files and directories.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      The following example shows a partitioned table that has associated statistics produced by the
+      <codeph>COMPUTE INCREMENTAL STATS</codeph> statement, and how the situation evolves as statistics are dropped
+      from specific partitions, then the entire table.
+    </p>
+
+    <p>
+      Initially, all table and column statistics are filled in.
+    </p>
+
+<!-- Note: chopped off any excess characters at position 87 and after,
+           to avoid weird wrapping in PDF.
+           Applies to any subsequent examples with output from SHOW ... STATS too. -->
+
+<codeblock>show table stats item_partitioned;
++-------------+-------+--------+----------+--------------+---------+-----------------
+| i_category  | #Rows | #Files | Size     | Bytes Cached | Format  | Incremental stats
++-------------+-------+--------+----------+--------------+---------+-----------------
+| Books       | 1733  | 1      | 223.74KB | NOT CACHED   | PARQUET | true
+| Children    | 1786  | 1      | 230.05KB | NOT CACHED   | PARQUET | true
+| Electronics | 1812  | 1      | 232.67KB | NOT CACHED   | PARQUET | true
+| Home        | 1807  | 1      | 232.56KB | NOT CACHED   | PARQUET | true
+| Jewelry     | 1740  | 1      | 223.72KB | NOT CACHED   | PARQUET | true
+| Men         | 1811  | 1      | 231.25KB | NOT CACHED   | PARQUET | true
+| Music       | 1860  | 1      | 237.90KB | NOT CACHED   | PARQUET | true
+| Shoes       | 1835  | 1      | 234.90KB | NOT CACHED   | PARQUET | true
+| Sports      | 1783  | 1      | 227.97KB | NOT CACHED   | PARQUET | true
+| Women       | 1790  | 1      | 226.27KB | NOT CACHED   | PARQUET | true
+| Total       | 17957 | 10     | 2.25MB   | 0B           |         |
++-------------+-------+--------+----------+--------------+---------+-----------------
+show column stats item_partitioned;
++------------------+-----------+------------------+--------+----------+--------------
+| Column           | Type      | #Distinct Values | #Nulls | Max Size | Avg Size
++------------------+-----------+------------------+--------+----------+--------------
+| i_item_sk        | INT       | 19443            | -1     | 4        | 4
+| i_item_id        | STRING    | 9025             | -1     | 16       | 16
+| i_rec_start_date | TIMESTAMP | 4                | -1     | 16       | 16
+| i_rec_end_date   | TIMESTAMP | 3                | -1     | 16       | 16
+| i_item_desc      | STRING    | 13330            | -1     | 200      | 100.302803039
+| i_current_price  | FLOAT     | 2807             | -1     | 4        | 4
+| i_wholesale_cost | FLOAT     | 2105             | -1     | 4        | 4
+| i_brand_id       | INT       | 965              | -1     | 4        | 4
+| i_brand          | STRING    | 725              | -1     | 22       | 16.1776008605
+| i_class_id       | INT       | 16               | -1     | 4        | 4
+| i_class          | STRING    | 101              | -1     | 15       | 7.76749992370
+| i_category_id    | INT       | 10               | -1     | 4        | 4
+| i_manufact_id    | INT       | 1857             | -1     | 4        | 4
+| i_manufact       | STRING    | 1028             | -1     | 15       | 11.3295001983
+| i_size           | STRING    | 8                | -1     | 11       | 4.33459997177
+| i_formulation    | STRING    | 12884            | -1     | 20       | 19.9799995422
+| i_color          | STRING    | 92               | -1     | 10       | 5.38089990615
+| i_units          | STRING    | 22               | -1     | 7        | 4.18690013885
+| i_container      | STRING    | 2                | -1     | 7        | 6.99259996414
+| i_manager_id     | INT       | 105              | -1     | 4        | 4
+| i_product_name   | STRING    | 19094            | -1     | 25       | 18.0233001708
+| i_category       | STRING    | 10               | 0      | -1       | -1
++------------------+-----------+------------------+--------+----------+--------------
+</codeblock>
+
+    <p>
+      To remove statistics for particular partitions, use the <codeph>DROP INCREMENTAL STATS</codeph> statement.
+      After removing statistics for two partitions, the table-level statistics reflect that change in the
+      <codeph>#Rows</codeph> and <codeph>Incremental stats</codeph> fields. The counts, maximums, and averages of
+      the column-level statistics are unaffected.
+    </p>
+
+    <note>
+      (It is possible that the row count might be preserved in future after a <codeph>DROP INCREMENTAL
+      STATS</codeph> statement. Check the resolution of the issue
+      <xref href="https://issues.cloudera.org/browse/IMPALA-1615" scope="external" format="html">IMPALA-1615</xref>.)
+    </note>
+
+<codeblock>drop incremental stats item_partitioned partition (i_category='Sports');
+drop incremental stats item_partitioned partition (i_category='Electronics');
+
+show table stats item_partitioned
++-------------+-------+--------+----------+--------------+---------+------------------
+| i_category  | #Rows | #Files | Size     | Bytes Cached | Format  | Incremental stats
++-------------+-------+--------+----------+--------------+---------+-----------------
+| Books       | 1733  | 1      | 223.74KB | NOT CACHED   | PARQUET | true
+| Children    | 1786  | 1      | 230.05KB | NOT CACHED   | PARQUET | true
+| Electronics | -1    | 1      | 232.67KB | NOT CACHED   | PARQUET | false
+| Home        | 1807  | 1      | 232.56KB | NOT CACHED   | PARQUET | true
+| Jewelry     | 1740  | 1      | 223.72KB | NOT CACHED   | PARQUET | true
+| Men         | 1811  | 1      | 231.25KB | NOT CACHED   | PARQUET | true
+| Music       | 1860  | 1      | 237.90KB | NOT CACHED   | PARQUET | true
+| Shoes       | 1835  | 1      | 234.90KB | NOT CACHED   | PARQUET | true
+| Sports      | -1    | 1      | 227.97KB | NOT CACHED   | PARQUET | false
+| Women       | 1790  | 1      | 226.27KB | NOT CACHED   | PARQUET | true
+| Total       | 17957 | 10     | 2.25MB   | 0B           |         |
++-------------+-------+--------+----------+--------------+---------+-----------------
+show column stats item_partitioned
++------------------+-----------+------------------+--------+----------+--------------
+| Column           | Type      | #Distinct Values | #Nulls | Max Size | Avg Size
++------------------+-----------+------------------+--------+----------+--------------
+| i_item_sk        | INT       | 19443            | -1     | 4        | 4
+| i_item_id        | STRING    | 9025             | -1     | 16       | 16
+| i_rec_start_date | TIMESTAMP | 4                | -1     | 16       | 16
+| i_rec_end_date   | TIMESTAMP | 3                | -1     | 16       | 16
+| i_item_desc      | STRING    | 13330            | -1     | 200      | 100.302803039
+| i_current_price  | FLOAT     | 2807             | -1     | 4        | 4
+| i_wholesale_cost | FLOAT     | 2105             | -1     | 4        | 4
+| i_brand_id       | INT       | 965              | -1     | 4        | 4
+| i_brand          | STRING    | 725              | -1     | 22       | 16.1776008605
+| i_class_id       | INT       | 16               | -1     | 4        | 4
+| i_class          | STRING    | 101              | -1     | 15       | 7.76749992370
+| i_category_id    | INT       | 10               | -1     | 4        | 4
+| i_manufact_id    | INT       | 1857             | -1     | 4        | 4
+| i_manufact       | STRING    | 1028             | -1     | 15       | 11.3295001983
+| i_size           | STRING    | 8                | -1     | 11       | 4.33459997177
+| i_formulation    | STRING    | 12884            | -1     | 20       | 19.9799995422
+| i_color          | STRING    | 92               | -1     | 10       | 5.38089990615
+| i_units          | STRING    | 22               | -1     | 7        | 4.18690013885
+| i_container      | STRING    | 2                | -1     | 7        | 6.99259996414
+| i_manager_id     | INT       | 105              | -1     | 4        | 4
+| i_product_name   | STRING    | 19094            | -1     | 25       | 18.0233001708
+| i_category       | STRING    | 10               | 0      | -1       | -1
++------------------+-----------+------------------+--------+----------+--------------
+</codeblock>
+
+    <p>
+      To remove all statistics from the table, whether produced by <codeph>COMPUTE STATS</codeph> or
+      <codeph>COMPUTE INCREMENTAL STATS</codeph>, use the <codeph>DROP STATS</codeph> statement without the
+      <codeph>INCREMENTAL</codeph> clause). Now, both table-level and column-level statistics are reset.
+    </p>
+
+<codeblock>drop stats item_partitioned;
+
+show table stats item_partitioned
++-------------+-------+--------+----------+--------------+---------+------------------
+| i_category  | #Rows | #Files | Size     | Bytes Cached | Format  | Incremental stats
++-------------+-------+--------+----------+--------------+---------+------------------
+| Books       | -1    | 1      | 223.74KB | NOT CACHED   | PARQUET | false
+| Children    | -1    | 1      | 230.05KB | NOT CACHED   | PARQUET | false
+| Electronics | -1    | 1      | 232.67KB | NOT CACHED   | PARQUET | false
+| Home        | -1    | 1      | 232.56KB | NOT CACHED   | PARQUET | false
+| Jewelry     | -1    | 1      | 223.72KB | NOT CACHED   | PARQUET | false
+| Men         | -1    | 1      | 231.25KB | NOT CACHED   | PARQUET | false
+| Music       | -1    | 1      | 237.90KB | NOT CACHED   | PARQUET | false
+| Shoes       | -1    | 1      | 234.90KB | NOT CACHED   | PARQUET | false
+| Sports      | -1    | 1      | 227.97KB | NOT CACHED   | PARQUET | false
+| Women       | -1    | 1      | 226.27KB | NOT CACHED   | PARQUET | false
+| Total       | -1    | 10     | 2.25MB   | 0B           |         |
++-------------+-------+--------+----------+--------------+---------+------------------
+show column stats item_partitioned
++------------------+-----------+------------------+--------+----------+----------+
+| Column           | Type      | #Distinct Values | #Nulls | Max Size | Avg Size |
++------------------+-----------+------------------+--------+----------+----------+
+| i_item_sk        | INT       | -1               | -1     | 4        | 4        |
+| i_item_id        | STRING    | -1               | -1     | -1       | -1       |
+| i_rec_start_date | TIMESTAMP | -1               | -1     | 16       | 16       |
+| i_rec_end_date   | TIMESTAMP | -1               | -1     | 16       | 16       |
+| i_item_desc      | STRING    | -1               | -1     | -1       | -1       |
+| i_current_price  | FLOAT     | -1               | -1     | 4        | 4        |
+| i_wholesale_cost | FLOAT     | -1               | -1     | 4        | 4        |
+| i_brand_id       | INT       | -1               | -1     | 4        | 4        |
+| i_brand          | STRING    | -1               | -1     | -1       | -1       |
+| i_class_id       | INT       | -1               | -1     | 4        | 4        |
+| i_class          | STRING    | -1               | -1     | -1       | -1       |
+| i_category_id    | INT       | -1               | -1     | 4        | 4        |
+| i_manufact_id    | INT       | -1               | -1     | 4        | 4        |
+| i_manufact       | STRING    | -1               | -1     | -1       | -1       |
+| i_size           | STRING    | -1               | -1     | -1       | -1       |
+| i_formulation    | STRING    | -1               | -1     | -1       | -1       |
+| i_color          | STRING    | -1               | -1     | -1       | -1       |
+| i_units          | STRING    | -1               | -1     | -1       | -1       |
+| i_container      | STRING    | -1               | -1     | -1       | -1       |
+| i_manager_id     | INT       | -1               | -1     | 4        | 4        |
+| i_product_name   | STRING    | -1               | -1     | -1       | -1       |
+| i_category       | STRING    | 10               | 0      | -1       | -1       |
++------------------+-----------+------------------+--------+----------+----------+
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_compute_stats.xml#compute_stats"/>, <xref href="impala_show.xml#show_table_stats"/>,
+      <xref href="impala_show.xml#show_column_stats"/>, <xref href="impala_perf_stats.xml#perf_stats"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_drop_table.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_drop_table.xml b/docs/topics/impala_drop_table.xml
new file mode 100644
index 0000000..8a75158
--- /dev/null
+++ b/docs/topics/impala_drop_table.xml
@@ -0,0 +1,150 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="drop_table">
+
+  <title>DROP TABLE Statement</title>
+  <titlealts audience="PDF"><navtitle>DROP TABLE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Schemas"/>
+      <data name="Category" value="S3"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">DROP TABLE statement</indexterm>
+      Removes an Impala table. Also removes the underlying HDFS data files for internal tables, although not for
+      external tables.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>DROP TABLE [IF EXISTS] [<varname>db_name</varname>.]<varname>table_name</varname> <ph rev="2.3.0">[PURGE]</ph></codeblock>
+
+    <p>
+      <b>IF EXISTS clause:</b>
+    </p>
+
+    <p>
+      The optional <codeph>IF EXISTS</codeph> clause makes the statement succeed whether or not the table exists.
+      If the table does exist, it is dropped; if it does not exist, the statement has no effect. This capability is
+      useful in standardized setup scripts that remove existing schema objects and create new ones. By using some
+      combination of <codeph>IF EXISTS</codeph> for the <codeph>DROP</codeph> statements and <codeph>IF NOT
+      EXISTS</codeph> clauses for the <codeph>CREATE</codeph> statements, the script can run successfully the first
+      time you run it (when the objects do not exist yet) and subsequent times (when some or all of the objects do
+      already exist).
+    </p>
+
+    <p rev="2.3.0">
+      <b>PURGE clause:</b>
+    </p>
+
+    <p rev="2.3.0"> The optional <codeph>PURGE</codeph> keyword, available in
+      <keyword keyref="impala23_full"/> and higher, causes Impala to remove the associated
+      HDFS data files immediately, rather than going through the HDFS trashcan
+      mechanism. Use this keyword when dropping a table if it is crucial to
+      remove the data as quickly as possible to free up space, or if there is a
+      problem with the trashcan, such as the trash cannot being configured or
+      being in a different HDFS encryption zone than the data files. </p>
+
+    <p conref="../shared/impala_common.xml#common/ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      By default, Impala removes the associated HDFS directory and data files for the table. If you issue a
+      <codeph>DROP TABLE</codeph> and the data files are not deleted, it might be for the following reasons:
+    </p>
+
+    <ul>
+      <li>
+        If the table was created with the
+        <codeph><xref href="impala_tables.xml#external_tables">EXTERNAL</xref></codeph> clause, Impala leaves all
+        files and directories untouched. Use external tables when the data is under the control of other Hadoop
+        components, and Impala is only used to query the data files from their original locations.
+      </li>
+
+      <li>
+        Impala might leave the data files behind unintentionally, if there is no HDFS location available to hold
+        the HDFS trashcan for the <codeph>impala</codeph> user. See
+        <xref href="impala_prereqs.xml#prereqs_account"/> for the procedure to set up the required HDFS home
+        directory.
+      </li>
+    </ul>
+
+    <p>
+      Make sure that you are in the correct database before dropping a table, either by issuing a
+      <codeph>USE</codeph> statement first or by using a fully qualified name
+      <codeph><varname>db_name</varname>.<varname>table_name</varname></codeph>.
+    </p>
+
+    <p>
+      If you intend to issue a <codeph>DROP DATABASE</codeph> statement, first issue <codeph>DROP TABLE</codeph>
+      statements to remove all the tables in that database.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>create database temporary;
+use temporary;
+create table unimportant (x int);
+create table trivial (s string);
+-- Drop a table in the current database.
+drop table unimportant;
+-- Switch to a different database.
+use default;
+-- To drop a table in a different database...
+drop table trivial;
+<i>ERROR: AnalysisException: Table does not exist: default.trivial</i>
+-- ...use a fully qualified name.
+drop table temporary.trivial;</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/disk_space_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/s3_blurb"/>
+    <p rev="2.6.0 CDH-39913 IMPALA-1878">
+      The <codeph>DROP TABLE</codeph> statement can remove data files from S3
+      if the associated S3 table is an internal table.
+      In <keyword keyref="impala26_full"/> and higher, as part of improved support for writing
+      to S3, Impala also removes the associated folder when dropping an internal table
+      that resides on S3.
+      See <xref href="impala_s3.xml#s3"/> for details about working with S3 tables.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/s3_drop_table_purge"/>
+
+    <p conref="../shared/impala_common.xml#common/s3_ddl"/>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+    <p rev="CDH-19187">
+      For an internal table, the user ID that the <cmdname>impalad</cmdname> daemon runs under,
+      typically the <codeph>impala</codeph> user, must have write
+      permission for all the files and directories that make up the table.
+    </p>
+    <p>
+      For an external table, dropping the table only involves changes to metadata in the metastore database.
+      Because Impala does not remove any HDFS files or directories when external tables are dropped,
+      no particular permissions are needed for the associated HDFS files or directories.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_tables.xml#tables"/>,
+      <xref href="impala_alter_table.xml#alter_table"/>, <xref href="impala_create_table.xml#create_table"/>,
+      <xref href="impala_partitioning.xml#partitioning"/>, <xref href="impala_tables.xml#internal_tables"/>,
+      <xref href="impala_tables.xml#external_tables"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_drop_view.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_drop_view.xml b/docs/topics/impala_drop_view.xml
new file mode 100644
index 0000000..627fd47
--- /dev/null
+++ b/docs/topics/impala_drop_view.xml
@@ -0,0 +1,49 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.1" id="drop_view">
+
+  <title>DROP VIEW Statement</title>
+  <titlealts audience="PDF"><navtitle>DROP VIEW</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="Schemas"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Views"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">DROP VIEW statement</indexterm>
+      Removes the specified view, which was originally created by the <codeph>CREATE VIEW</codeph> statement.
+      Because a view is purely a logical construct (an alias for a query) with no physical data behind it,
+      <codeph>DROP VIEW</codeph> only involves changes to metadata in the metastore database, not any data files in
+      HDFS.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>DROP VIEW [IF EXISTS] [<varname>db_name</varname>.]<varname>view_name</varname></codeblock>
+
+    <p conref="../shared/impala_common.xml#common/ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/create_drop_view_examples"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_views.xml#views"/>, <xref href="impala_create_view.xml#create_view"/>,
+      <xref href="impala_alter_view.xml#alter_view"/>
+    </p>
+  </conbody>
+</concept>


[29/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_file_formats.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_file_formats.xml b/docs/topics/impala_file_formats.xml
new file mode 100644
index 0000000..48b9e7c
--- /dev/null
+++ b/docs/topics/impala_file_formats.xml
@@ -0,0 +1,270 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="file_formats">
+
+  <title>How Impala Works with Hadoop File Formats</title>
+  <titlealts audience="PDF"><navtitle>File Formats</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Concepts"/>
+      <data name="Category" value="Hadoop"/>
+      <data name="Category" value="File Formats"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+      <!-- Like Impala Administration, this page has a fair bit of info already, but it could benefit from wiki-style embedded of intro text from those other pages. -->
+      <!-- In this case, that would also enable a good in-page TOC since there is already one lonely subtopic on this same page. -->
+      <data name="Category" value="Stub Pages"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">file formats</indexterm>
+      <indexterm audience="Cloudera">compression</indexterm>
+      Impala supports several familiar file formats used in Apache Hadoop. Impala can load and query data files
+      produced by other Hadoop components such as Pig or MapReduce, and data files produced by Impala can be used
+      by other components also. The following sections discuss the procedures, limitations, and performance
+      considerations for using each file format with Impala.
+    </p>
+
+    <p>
+      The file format used for an Impala table has significant performance consequences. Some file formats include
+      compression support that affects the size of data on the disk and, consequently, the amount of I/O and CPU
+      resources required to deserialize data. The amounts of I/O and CPU resources required can be a limiting
+      factor in query performance since querying often begins with moving and decompressing data. To reduce the
+      potential impact of this part of the process, data is often compressed. By compressing data, a smaller total
+      number of bytes are transferred from disk to memory. This reduces the amount of time taken to transfer the
+      data, but a tradeoff occurs when the CPU decompresses the content.
+    </p>
+
+    <p>
+      Impala can query files encoded with most of the popular file formats and compression codecs used in Hadoop.
+      Impala can create and insert data into tables that use some file formats but not others; for file formats
+      that Impala cannot write to, create the table in Hive, issue the <codeph>INVALIDATE METADATA <varname>table_name</varname></codeph>
+      statement in <codeph>impala-shell</codeph>, and query the table through Impala. File formats can be
+      structured, in which case they may include metadata and built-in compression. Supported formats include:
+    </p>
+
+    <table>
+      <title>File Format Support in Impala</title>
+      <tgroup cols="5">
+        <colspec colname="1" colwidth="10*"/>
+        <colspec colname="2" colwidth="10*"/>
+        <colspec colname="3" colwidth="20*"/>
+        <colspec colname="4" colwidth="30*"/>
+        <colspec colname="5" colwidth="30*"/>
+        <thead>
+          <row>
+            <entry>
+              File Type
+            </entry>
+            <entry>
+              Format
+            </entry>
+            <entry>
+              Compression Codecs
+            </entry>
+            <entry>
+              Impala Can CREATE?
+            </entry>
+            <entry>
+              Impala Can INSERT?
+            </entry>
+          </row>
+        </thead>
+        <tbody>
+          <row id="parquet_support">
+            <entry>
+              <xref href="impala_parquet.xml#parquet">Parquet</xref>
+            </entry>
+            <entry>
+              Structured
+            </entry>
+            <entry>
+              Snappy, gzip; currently Snappy by default
+            </entry>
+            <entry>
+              Yes.
+            </entry>
+            <entry>
+              Yes: <codeph>CREATE TABLE</codeph>, <codeph>INSERT</codeph>, <codeph>LOAD DATA</codeph>, and query.
+            </entry>
+          </row>
+          <row id="txtfile_support">
+            <entry>
+              <xref href="impala_txtfile.xml#txtfile">Text</xref>
+            </entry>
+            <entry>
+              Unstructured
+            </entry>
+            <entry rev="2.0.0">
+              LZO, gzip, bzip2, Snappy
+            </entry>
+            <entry>
+              Yes. For <codeph>CREATE TABLE</codeph> with no <codeph>STORED AS</codeph> clause, the default file
+              format is uncompressed text, with values separated by ASCII <codeph>0x01</codeph> characters
+              (typically represented as Ctrl-A).
+            </entry>
+            <entry>
+              Yes: <codeph>CREATE TABLE</codeph>, <codeph>INSERT</codeph>, <codeph>LOAD DATA</codeph>, and query.
+              If LZO compression is used, you must create the table and load data in Hive. If other kinds of
+              compression are used, you must load data through <codeph>LOAD DATA</codeph>, Hive, or manually in
+              HDFS.
+
+<!--            <ph rev="2.0.0">Impala 2.0 and higher can write LZO-compressed text data; for earlier Impala releases,  you must create the table and load data in Hive.</ph> -->
+            </entry>
+          </row>
+          <row id="avro_support">
+            <entry>
+              <xref href="impala_avro.xml#avro">Avro</xref>
+            </entry>
+            <entry>
+              Structured
+            </entry>
+            <entry>
+              Snappy, gzip, deflate, bzip2
+            </entry>
+            <entry rev="1.4.0">
+              Yes, in Impala 1.4.0 and higher. Before that, create the table using Hive.
+            </entry>
+            <entry>
+              No. Import data by using <codeph>LOAD DATA</codeph> on data files already in the right format, or use
+              <codeph>INSERT</codeph> in Hive followed by <codeph>REFRESH <varname>table_name</varname></codeph> in Impala.
+            </entry>
+<!-- <entry rev="2.0.0">Yes, in Impala 2.0 and higher. For earlier Impala releases, load data through <codeph>LOAD DATA</codeph> on data files already in the right format, or use <codeph>INSERT</codeph> in Hive.</entry> -->
+          </row>
+          <row id="rcfile_support">
+            <entry>
+              <xref href="impala_rcfile.xml#rcfile">RCFile</xref>
+            </entry>
+            <entry>
+              Structured
+            </entry>
+            <entry>
+              Snappy, gzip, deflate, bzip2
+            </entry>
+            <entry>
+              Yes.
+            </entry>
+            <entry>
+              No. Import data by using <codeph>LOAD DATA</codeph> on data files already in the right format, or use
+              <codeph>INSERT</codeph> in Hive followed by <codeph>REFRESH <varname>table_name</varname></codeph> in Impala.
+            </entry>
+<!--
+            <entry rev="2.0.0">Yes, in Impala 2.0 and higher. For earlier Impala releases, load data through <codeph>LOAD DATA</codeph> on data files already in the right format, or use <codeph>INSERT</codeph> in Hive.</entry>
+            -->
+          </row>
+          <row id="sequencefile_support">
+            <entry>
+              <xref href="impala_seqfile.xml#seqfile">SequenceFile</xref>
+            </entry>
+            <entry>
+              Structured
+            </entry>
+            <entry>
+              Snappy, gzip, deflate, bzip2
+            </entry>
+            <entry>Yes.</entry>
+            <entry>
+              No. Import data by using <codeph>LOAD DATA</codeph> on data files already in the right format, or use
+              <codeph>INSERT</codeph> in Hive followed by <codeph>REFRESH <varname>table_name</varname></codeph> in Impala.
+            </entry>
+<!--
+            <entry rev="2.0.0">
+              Yes, in Impala 2.0 and higher. For earlier Impala releases, load data through <codeph>LOAD
+              DATA</codeph> on data files already in the right format, or use <codeph>INSERT</codeph> in Hive.
+            </entry>
+-->
+          </row>
+        </tbody>
+      </tgroup>
+    </table>
+
+    <p rev="DOCS-1370">
+      Impala can only query the file formats listed in the preceding table.
+      In particular, Impala does not support the ORC file format.
+    </p>
+
+    <p>
+      Impala supports the following compression codecs:
+    </p>
+
+    <ul>
+      <li rev="2.0.0">
+        Snappy. Recommended for its effective balance between compression ratio and decompression speed. Snappy
+        compression is very fast, but gzip provides greater space savings. Supported for text files in Impala 2.0
+        and higher.
+<!-- Not supported for text files. -->
+      </li>
+
+      <li rev="2.0.0">
+        Gzip. Recommended when achieving the highest level of compression (and therefore greatest disk-space
+        savings) is desired. Supported for text files in Impala 2.0 and higher.
+      </li>
+
+      <li>
+        Deflate. Not supported for text files.
+      </li>
+
+      <li rev="2.0.0">
+        Bzip2. Supported for text files in Impala 2.0 and higher.
+<!-- Not supported for text files. -->
+      </li>
+
+      <li>
+        <p rev="2.0.0"> LZO, for text files only. Impala can query
+          LZO-compressed text tables, but currently cannot create them or insert
+          data into them; perform these operations in Hive. </p>
+      </li>
+    </ul>
+  </conbody>
+
+  <concept id="file_format_choosing">
+
+    <title>Choosing the File Format for a Table</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Planning"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        Different file formats and compression codecs work better for different data sets. While Impala typically
+        provides performance gains regardless of file format, choosing the proper format for your data can yield
+        further performance improvements. Use the following considerations to decide which combination of file
+        format and compression to use for a particular table:
+      </p>
+
+      <ul>
+        <li>
+          If you are working with existing files that are already in a supported file format, use the same format
+          for the Impala table where practical. If the original format does not yield acceptable query performance
+          or resource usage, consider creating a new Impala table with different file format or compression
+          characteristics, and doing a one-time conversion by copying the data to the new table using the
+          <codeph>INSERT</codeph> statement. Depending on the file format, you might run the
+          <codeph>INSERT</codeph> statement in <codeph>impala-shell</codeph> or in Hive.
+        </li>
+
+        <li>
+          Text files are convenient to produce through many different tools, and are human-readable for ease of
+          verification and debugging. Those characteristics are why text is the default format for an Impala
+          <codeph>CREATE TABLE</codeph> statement. When performance and resource usage are the primary
+          considerations, use one of the other file formats and consider using compression. A typical workflow
+          might involve bringing data into an Impala table by copying CSV or TSV files into the appropriate data
+          directory, and then using the <codeph>INSERT ... SELECT</codeph> syntax to copy the data into a table
+          using a different, more compact file format.
+        </li>
+
+        <li>
+          If your architecture involves storing data to be queried in memory, do not compress the data. There is no
+          I/O savings since the data does not need to be moved from disk, but there is a CPU cost to decompress the
+          data.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+</concept>


[45/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_analytic_functions.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_analytic_functions.xml b/docs/topics/impala_analytic_functions.xml
new file mode 100644
index 0000000..0a31363
--- /dev/null
+++ b/docs/topics/impala_analytic_functions.xml
@@ -0,0 +1,1739 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="2.0.0" id="analytic_functions">
+
+  <title>Impala Analytic Functions</title>
+
+  <titlealts audience="PDF">
+
+    <navtitle>Analytic Functions</navtitle>
+
+  </titlealts>
+
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="Aggregate Functions"/>
+      <data name="Category" value="Analytic Functions"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Querying"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.0.0">
+      <indexterm audience="Cloudera">analytic functions</indexterm>
+
+      <indexterm audience="Cloudera">window functions</indexterm>
+      Analytic functions (also known as window functions) are a special category of built-in functions. Like
+      aggregate functions, they examine the contents of multiple input rows to compute each output value. However,
+      rather than being limited to one result value per <codeph>GROUP BY</codeph> group, they operate on
+      <term>windows</term> where the input rows are ordered and grouped using flexible conditions expressed through
+      an <codeph>OVER()</codeph> clause.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/added_in_20"/>
+
+<!--
+    <p>
+      Analytic functions produce one output value for each input row, like scalar functions such as
+      <codeph>length()</codeph> or
+      <codeph>substr()</codeph>.
+    </p>
+-->
+
+    <p>
+      Some functions, such as <codeph>LAG()</codeph> and <codeph>RANK()</codeph>, can only be used in this analytic
+      context. Some aggregate functions do double duty: when you call the aggregation functions such as
+      <codeph>MAX()</codeph>, <codeph>SUM()</codeph>, <codeph>AVG()</codeph>, and so on with an
+      <codeph>OVER()</codeph> clause, they produce an output value for each row, based on computations across other
+      rows in the window.
+    </p>
+
+    <p>
+      Although analytic functions often compute the same value you would see from an aggregate function in a
+      <codeph>GROUP BY</codeph> query, the analytic functions produce a value for each row in the result set rather
+      than a single value for each group. This flexibility lets you include additional columns in the
+      <codeph>SELECT</codeph> list, offering more opportunities for organizing and filtering the result set.
+    </p>
+
+    <p>
+      Analytic function calls are only allowed in the <codeph>SELECT</codeph> list and in the outermost
+      <codeph>ORDER BY</codeph> clause of the query. During query processing, analytic functions are evaluated
+      after other query stages such as joins, <codeph>WHERE</codeph>, and <codeph>GROUP BY</codeph>,
+    </p>
+
+<!-- Oracle doesn't show examples until it gets to the actual functions, so let's follow that lead.
+    <p>
+      The following example shows a very simple call to <codeph>MAX()</codeph> in
+      an analytic context, and a similar query using a <codeph>GROUP BY</codeph> clause.
+    </p>
+-->
+
+<!--
+This basic query could be represented either as an analytic
+function call or an aggregation function call in a <codeph>GROUP BY</codeph> query.
+For more elaborate kinds of computations, the flexibility of the analytic window
+makes that the preferred option.
+-->
+
+<!-- TK: construct sample data and fill in query results. -->
+
+<!-- Other DB docs don't necessarily include examples up at this level, only for the individual functions.
+     So maybe take these placeholders out entirely.
+
+<codeblock>SELECT year, month, max(degrees_c) OVER (PARTITION BY year) FROM historical_temps;
+SELECT year, month, max(degrees_c) FROM historical_temps GROUP BY year;
+</codeblock>
+-->
+
+    <p>
+      The rows that are part of each partition are analyzed by computations across an ordered or unordered set of
+      rows. For example, <codeph>COUNT()</codeph> and <codeph>SUM()</codeph> might be applied to all the rows in
+      the partition, in which case the order of analysis does not matter. The <codeph>ORDER BY</codeph> clause
+      might be used inside the <codeph>OVER()</codeph> clause to defines the ordering that applies to functions
+      such as <codeph>LAG()</codeph> and <codeph>FIRST_VALUE()</codeph>.
+    </p>
+
+<!-- TK: output needed here also. -->
+
+<!--
+<codeblock>SELECT year, month, max(degrees_c) OVER (PARTITION BY year ORDER BY MONTH DESC) FROM historical_temps;
+</codeblock>
+-->
+
+    <p>
+      Analytic functions are frequently used in fields such as finance and science to provide trend, outlier, and
+      bucketed analysis for large data sets. You might also see the term <q>window functions</q> in database
+      literature, referring to the sequence of rows (the <q>window</q>) that the function call applies to,
+      particularly when the <codeph>OVER</codeph> clause includes a <codeph>ROWS</codeph> or <codeph>RANGE</codeph>
+      keyword.
+    </p>
+
+    <p>
+      The following sections describe the analytic query clauses and the pure analytic functions provided by
+      Impala. For usage information about aggregate functions in an analytic context, see
+      <xref href="impala_aggregate_functions.xml#aggregate_functions"/>.
+    </p>
+
+    <p outputclass="toc inpage"/>
+
+  </conbody>
+
+  <concept id="over">
+
+    <title>OVER Clause</title>
+
+    <conbody>
+
+      <p>
+        The <codeph>OVER</codeph> clause is required for calls to pure analytic functions such as
+        <codeph>LEAD()</codeph>, <codeph>RANK()</codeph>, and <codeph>FIRST_VALUE()</codeph>. When you include an
+        <codeph>OVER</codeph> clause with calls to aggregate functions such as <codeph>MAX()</codeph>,
+        <codeph>COUNT()</codeph>, or <codeph>SUM()</codeph>, they operate as analytic functions.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>function(<varname>args</varname>) OVER([<varname>partition_by_clause</varname>] [<varname>order_by_clause</varname> [<varname>window_clause</varname>]])
+
+partition_by_clause ::= PARTITION BY <varname>expr</varname> [, <varname>expr</varname> ...]
+order_by_clause ::= ORDER BY <varname>expr</varname>  [ASC | DESC] [NULLS FIRST | NULLS LAST] [, <varname>expr</varname> [ASC | DESC] [NULLS FIRST | NULLS LAST] ...]
+window_clause: See <xref href="#window_clause">Window Clause</xref>
+</codeblock>
+
+      <p>
+        <b>PARTITION BY clause:</b>
+      </p>
+
+      <p>
+        The <codeph>PARTITION BY</codeph> clause acts much like the <codeph>GROUP BY</codeph> clause in the
+        outermost block of a query. It divides the rows into groups containing identical values in one or more
+        columns. These logical groups are known as <term>partitions</term>. Throughout the discussion of analytic
+        functions, <q>partitions</q> refers to the groups produced by the <codeph>PARTITION BY</codeph> clause, not
+        to partitioned tables. However, note the following limitation that applies specifically to analytic function
+        calls involving partitioned tables.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/analytic_partition_pruning_caveat"/>
+
+      <p>
+        The sequence of results from an analytic function <q>resets</q> for each new partition in the result set.
+        That is, the set of preceding or following rows considered by the analytic function always come from a
+        single partition. Any <codeph>MAX()</codeph>, <codeph>SUM()</codeph>, <codeph>ROW_NUMBER()</codeph>, and so
+        on apply to each partition independently. Omit the <codeph>PARTITION BY</codeph> clause to apply the
+        analytic operation to all the rows in the table.
+      </p>
+
+      <p>
+        <b>ORDER BY clause:</b>
+      </p>
+
+      <p>
+        The <codeph>ORDER BY</codeph> clause works much like the <codeph>ORDER BY</codeph> clause in the outermost
+        block of a query. It defines the order in which rows are evaluated for the entire input set, or for each
+        group produced by a <codeph>PARTITION BY</codeph> clause. You can order by one or multiple expressions, and
+        for each expression optionally choose ascending or descending order and whether nulls come first or last in
+        the sort order. Because this <codeph>ORDER BY</codeph> clause only defines the order in which rows are
+        evaluated, if you want the results to be output in a specific order, also include an <codeph>ORDER
+        BY</codeph> clause in the outer block of the query.
+      </p>
+
+      <p>
+        When the <codeph>ORDER BY</codeph> clause is omitted, the analytic function applies to all items in the
+        group produced by the <codeph>PARTITION BY</codeph> clause. When the <codeph>ORDER BY</codeph> clause is
+        included, the analysis can apply to all or a subset of the items in the group, depending on the optional
+        window clause.
+      </p>
+
+      <p>
+        The order in which the rows are analyzed is only defined for those columns specified in <codeph>ORDER
+        BY</codeph> clauses.
+      </p>
+
+      <p>
+        One difference between the analytic and outer uses of the <codeph>ORDER BY</codeph> clause: inside the
+        <codeph>OVER</codeph> clause, <codeph>ORDER BY 1</codeph> or other integer value is interpreted as a
+        constant sort value (effectively a no-op) rather than referring to column 1.
+      </p>
+
+      <p>
+        <b>Window clause:</b>
+      </p>
+
+      <p>
+        The window clause is only allowed in combination with an <codeph>ORDER BY</codeph> clause. If the
+        <codeph>ORDER BY</codeph> clause is specified but the window clause is not, the default window is
+        <codeph>RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW</codeph>. See
+        <xref href="impala_analytic_functions.xml#window_clause"/> for full details.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/hbase_blurb"/>
+
+      <p>
+        Because HBase tables are optimized for single-row lookups rather than full scans, analytic functions using
+        the <codeph>OVER()</codeph> clause are not recommended for HBase tables. Although such queries work, their
+        performance is lower than on comparable tables using HDFS data files.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/parquet_blurb"/>
+
+      <p>
+        Analytic functions are very efficient for Parquet tables. The data that is examined during evaluation of
+        the <codeph>OVER()</codeph> clause comes from a specified set of columns, and the values for each column
+        are arranged sequentially within each data file.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/text_blurb"/>
+
+      <p>
+        Analytic functions are convenient to use with text tables for exploratory business intelligence. When the
+        volume of data is substantial, prefer to use Parquet tables for performance-critical analytic queries.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/added_in_20"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following example shows how to synthesize a numeric sequence corresponding to all the rows in a table.
+        The new table has the same columns as the old one, plus an additional column <codeph>ID</codeph> containing
+        the integers 1, 2, 3, and so on, corresponding to the order of a <codeph>TIMESTAMP</codeph> column in the
+        original table.
+      </p>
+
+<!-- TK: synthesize some data and fill in output here. -->
+
+<codeblock>CREATE TABLE events_with_id AS
+  SELECT
+    row_number() OVER (ORDER BY date_and_time) AS id,
+    c1, c2, c3, c4
+  FROM events;
+</codeblock>
+
+      <p>
+        The following example shows how to determine the number of rows containing each value for a column. Unlike
+        a corresponding <codeph>GROUP BY</codeph> query, this one can analyze a single column and still return all
+        values (not just the distinct ones) from the other columns.
+      </p>
+
+<!-- TK: verify the 'unbounded' shortcut syntax. -->
+
+<codeblock>SELECT x, y, z,
+  count() OVER (PARTITION BY x) AS how_many_x
+FROM t1;
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+      <p>
+        You cannot directly combine the <codeph>DISTINCT</codeph> operator with analytic function calls. You can
+        put the analytic function call in a <codeph>WITH</codeph> clause or an inline view, and apply the
+        <codeph>DISTINCT</codeph> operator to its result set.
+      </p>
+
+<codeblock>WITH t1 AS (SELECT x, sum(x) OVER (PARTITION BY x) AS total FROM t1)
+  SELECT DISTINCT x, total FROM t1;
+</codeblock>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="window_clause">
+
+    <title>Window Clause</title>
+
+    <conbody>
+
+      <p>
+        Certain analytic functions accept an optional <term>window clause</term>, which makes the function analyze
+        only certain rows <q>around</q> the current row rather than all rows in the partition. For example, you can
+        get a moving average by specifying some number of preceding and following rows, or a running count or
+        running total by specifying all rows up to the current position. This clause can result in different
+        analytic results for rows within the same partition.
+      </p>
+
+      <p>
+        The window clause is supported with the <codeph>AVG()</codeph>, <codeph>COUNT()</codeph>,
+        <codeph>FIRST_VALUE()</codeph>, <codeph>LAST_VALUE()</codeph>, and <codeph>SUM()</codeph> functions.
+<!-- To do: fill in this factoid under MAX and MIN also. -->
+        For <codeph>MAX()</codeph> and <codeph>MIN()</codeph>, the window clause only allowed if the start bound is
+        <codeph>UNBOUNDED PRECEDING</codeph>
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>ROWS BETWEEN [ { <varname>m</varname> | UNBOUNDED } PRECEDING | CURRENT ROW] [ AND [CURRENT ROW | { UNBOUNDED | <varname>n</varname> } FOLLOWING] ]
+RANGE BETWEEN [ {<varname>m</varname> | UNBOUNDED } PRECEDING | CURRENT ROW] [ AND [CURRENT ROW | { UNBOUNDED | <varname>n</varname> } FOLLOWING] ]</codeblock>
+
+      <p>
+        <codeph>ROWS BETWEEN</codeph> defines the size of the window in terms of the indexes of the rows in the
+        result set. The size of the window is predictable based on the clauses the position within the result set.
+      </p>
+
+      <p>
+        <codeph>RANGE BETWEEN</codeph> does not currently support numeric arguments to define a variable-size
+        sliding window.
+<!--
+Currently, it effectively works the same as the
+equivalent <codeph>ROWS BETWEEN</codeph> clause.
+-->
+      </p>
+
+<!--
+<p>
+<codeph>RANGE BETWEEN</codeph> defines the size of the window based on arithmetic comparisons
+of the values in the result set.
+The size of the window varies depending on the order and distribution of values.
+</p>
+-->
+
+      <p>
+        Currently, Impala supports only some combinations of arguments to the <codeph>RANGE</codeph> clause:
+      </p>
+
+      <ul>
+        <li>
+          <codeph>RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW</codeph> (the default when <codeph>ORDER
+          BY</codeph> is specified and the window clause is omitted)
+        </li>
+
+        <li>
+          <codeph>RANGE BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING</codeph>
+        </li>
+
+        <li>
+          <codeph>RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING</codeph>
+        </li>
+      </ul>
+
+      <p>
+        When <codeph>RANGE</codeph> is used, <codeph>CURRENT ROW</codeph> includes not just the current row but all
+        rows that are tied with the current row based on the <codeph>ORDER BY</codeph> expressions.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/added_in_20"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following examples show financial data for a fictional stock symbol <codeph>JDR</codeph>. The closing
+        price moves up and down each day.
+      </p>
+
+<codeblock>create table stock_ticker (stock_symbol string, closing_price decimal(8,2), closing_date timestamp);
+...load some data...
+select * from stock_ticker order by stock_symbol, closing_date
++--------------+---------------+---------------------+
+| stock_symbol | closing_price | closing_date        |
++--------------+---------------+---------------------+
+| JDR          | 12.86         | 2014-10-02 00:00:00 |
+| JDR          | 12.89         | 2014-10-03 00:00:00 |
+| JDR          | 12.94         | 2014-10-04 00:00:00 |
+| JDR          | 12.55         | 2014-10-05 00:00:00 |
+| JDR          | 14.03         | 2014-10-06 00:00:00 |
+| JDR          | 14.75         | 2014-10-07 00:00:00 |
+| JDR          | 13.98         | 2014-10-08 00:00:00 |
++--------------+---------------+---------------------+
+</codeblock>
+
+      <p>
+        The queries use analytic functions with window clauses to compute moving averages of the closing price. For
+        example, <codeph>ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING</codeph> produces an average of the value from a
+        3-day span, producing a different value for each row. The first row, which has no preceding row, only gets
+        averaged with the row following it. If the table contained more than one stock symbol, the
+        <codeph>PARTITION BY</codeph> clause would limit the window for the moving average to only consider the
+        prices for a single stock.
+      </p>
+
+<codeblock>select stock_symbol, closing_date, closing_price,
+  avg(closing_price) over (partition by stock_symbol order by closing_date
+    rows between 1 preceding and 1 following) as moving_average
+  from stock_ticker;
++--------------+---------------------+---------------+----------------+
+| stock_symbol | closing_date        | closing_price | moving_average |
++--------------+---------------------+---------------+----------------+
+| JDR          | 2014-10-02 00:00:00 | 12.86         | 12.87          |
+| JDR          | 2014-10-03 00:00:00 | 12.89         | 12.89          |
+| JDR          | 2014-10-04 00:00:00 | 12.94         | 12.79          |
+| JDR          | 2014-10-05 00:00:00 | 12.55         | 13.17          |
+| JDR          | 2014-10-06 00:00:00 | 14.03         | 13.77          |
+| JDR          | 2014-10-07 00:00:00 | 14.75         | 14.25          |
+| JDR          | 2014-10-08 00:00:00 | 13.98         | 14.36          |
++--------------+---------------------+---------------+----------------+
+</codeblock>
+
+      <p>
+        The clause <codeph>ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW</codeph> produces a cumulative moving
+        average, from the earliest data up to the value for each day.
+      </p>
+
+<codeblock>select stock_symbol, closing_date, closing_price,
+  avg(closing_price) over (partition by stock_symbol order by closing_date
+    rows between unbounded preceding and current row) as moving_average
+  from stock_ticker;
++--------------+---------------------+---------------+----------------+
+| stock_symbol | closing_date        | closing_price | moving_average |
++--------------+---------------------+---------------+----------------+
+| JDR          | 2014-10-02 00:00:00 | 12.86         | 12.86          |
+| JDR          | 2014-10-03 00:00:00 | 12.89         | 12.87          |
+| JDR          | 2014-10-04 00:00:00 | 12.94         | 12.89          |
+| JDR          | 2014-10-05 00:00:00 | 12.55         | 12.81          |
+| JDR          | 2014-10-06 00:00:00 | 14.03         | 13.05          |
+| JDR          | 2014-10-07 00:00:00 | 14.75         | 13.33          |
+| JDR          | 2014-10-08 00:00:00 | 13.98         | 13.42          |
++--------------+---------------------+---------------+----------------+
+</codeblock>
+
+<!-- Matt suggests not always true depending on data. Hiding until I can try myself.
+<p>
+The clause <codeph>RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW</codeph> would produce the same
+output as above. Because <codeph>RANGE</codeph> currently does not support numeric offsets while
+<codeph>ROWS</codeph> does, currently the <codeph>ROWS</codeph> syntax is more flexible.
+</p>
+-->
+
+    </conbody>
+
+  </concept>
+
+  <concept id="avg_analytic">
+
+    <title>AVG Function - Analytic Context</title>
+
+    <conbody>
+
+      <p>
+        You can include an <codeph>OVER</codeph> clause with a call to this function to use it as an analytic
+        function. See <xref href="impala_avg.xml#avg"/> for details and examples.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="count_analytic">
+
+    <title>COUNT Function - Analytic Context</title>
+
+    <conbody>
+
+      <p>
+        You can include an <codeph>OVER</codeph> clause with a call to this function to use it as an analytic
+        function. See <xref href="impala_count.xml#count"/> for details and examples.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.3.0" id="cume_dist">
+
+    <title>CUME_DIST Function (<keyword keyref="impala23"/> or higher only)</title>
+
+    <conbody>
+
+      <p>
+        Returns the cumulative distribution of a value. The value for each row in the result set is greater than 0
+        and less than or equal to 1.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>CUME_DIST (<varname>expr</varname>)
+  OVER ([<varname>partition_by_clause</varname>] <varname>order_by_clause</varname>)
+</codeblock>
+
+      <p>
+        The <codeph>ORDER BY</codeph> clause is required. The <codeph>PARTITION BY</codeph> clause is optional. The
+        window clause is not allowed.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p>
+        Within each partition of the result set, the <codeph>CUME_DIST()</codeph> value represents an ascending
+        sequence that ends at 1. Each value represents the proportion of rows in the partition whose values are
+        less than or equal to the value in the current row.
+      </p>
+
+      <p>
+        If the sequence of input values contains ties, the <codeph>CUME_DIST()</codeph> results are identical for the
+        tied values.
+      </p>
+
+      <p>
+        Impala only supports the <codeph>CUME_DIST()</codeph> function in an analytic context, not as a regular
+        aggregate function.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        This example uses a table with 9 rows. The <codeph>CUME_DIST()</codeph>
+        function evaluates the entire table because there is no <codeph>PARTITION BY</codeph> clause,
+        with the rows ordered by the weight of the animal.
+        the sequence of values shows that 1/9 of the values are less than or equal to the lightest
+        animal (mouse), 2/9 of the values are less than or equal to the second-lightest animal,
+        and so on up to the heaviest animal (elephant), where 9/9 of the rows are less than or
+        equal to its weight.
+      </p>
+
+<codeblock>create table animals (name string, kind string, kilos decimal(9,3));
+insert into animals values
+  ('Elephant', 'Mammal', 4000), ('Giraffe', 'Mammal', 1200), ('Mouse', 'Mammal', 0.020),
+  ('Condor', 'Bird', 15), ('Horse', 'Mammal', 500), ('Owl', 'Bird', 2.5),
+  ('Ostrich', 'Bird', 145), ('Polar bear', 'Mammal', 700), ('Housecat', 'Mammal', 5);
+
+select name, cume_dist() over (order by kilos) from animals;
++------------+-----------------------+
+| name       | cume_dist() OVER(...) |
++------------+-----------------------+
+| Elephant   | 1                     |
+| Giraffe    | 0.8888888888888888    |
+| Polar bear | 0.7777777777777778    |
+| Horse      | 0.6666666666666666    |
+| Ostrich    | 0.5555555555555556    |
+| Condor     | 0.4444444444444444    |
+| Housecat   | 0.3333333333333333    |
+| Owl        | 0.2222222222222222    |
+| Mouse      | 0.1111111111111111    |
++------------+-----------------------+
+</codeblock>
+
+      <p>
+        Using a <codeph>PARTITION BY</codeph> clause produces a separate sequence for each partition
+        group, in this case one for mammals and one for birds. Because there are 3 birds and 6 mammals,
+        the sequence illustrates how 1/3 of the <q>Bird</q> rows have a <codeph>kilos</codeph> value that is less than or equal to
+        the lightest bird, 1/6 of the <q>Mammal</q> rows have a <codeph>kilos</codeph> value that is less than or equal to
+        the lightest mammal, and so on until both the heaviest bird and heaviest mammal have a <codeph>CUME_DIST()</codeph>
+        value of 1.
+      </p>
+
+<codeblock>select name, kind, cume_dist() over (partition by kind order by kilos) from animals
++------------+--------+-----------------------+
+| name       | kind   | cume_dist() OVER(...) |
++------------+--------+-----------------------+
+| Ostrich    | Bird   | 1                     |
+| Condor     | Bird   | 0.6666666666666666    |
+| Owl        | Bird   | 0.3333333333333333    |
+| Elephant   | Mammal | 1                     |
+| Giraffe    | Mammal | 0.8333333333333334    |
+| Polar bear | Mammal | 0.6666666666666666    |
+| Horse      | Mammal | 0.5                   |
+| Housecat   | Mammal | 0.3333333333333333    |
+| Mouse      | Mammal | 0.1666666666666667    |
++------------+--------+-----------------------+
+</codeblock>
+
+      <p>
+        We can reverse the ordering within each partition group by using an <codeph>ORDER BY ... DESC</codeph>
+        clause within the <codeph>OVER()</codeph> clause. Now the lightest (smallest value of <codeph>kilos</codeph>)
+        animal of each kind has a <codeph>CUME_DIST()</codeph> value of 1.
+      </p>
+
+<codeblock>select name, kind, cume_dist() over (partition by kind order by kilos desc) from animals
++------------+--------+-----------------------+
+| name       | kind   | cume_dist() OVER(...) |
++------------+--------+-----------------------+
+| Owl        | Bird   | 1                     |
+| Condor     | Bird   | 0.6666666666666666    |
+| Ostrich    | Bird   | 0.3333333333333333    |
+| Mouse      | Mammal | 1                     |
+| Housecat   | Mammal | 0.8333333333333334    |
+| Horse      | Mammal | 0.6666666666666666    |
+| Polar bear | Mammal | 0.5                   |
+| Giraffe    | Mammal | 0.3333333333333333    |
+| Elephant   | Mammal | 0.1666666666666667    |
++------------+--------+-----------------------+
+</codeblock>
+
+      <p>
+        The following example manufactures some rows with identical values in the <codeph>kilos</codeph> column,
+        to demonstrate how the results look in case of tie values. For simplicity, it only shows the <codeph>CUME_DIST()</codeph>
+        sequence for the <q>Bird</q> rows. Now with 3 rows all with a value of 15, all of those rows have the same
+        <codeph>CUME_DIST()</codeph> value. 4/5 of the rows have a value for <codeph>kilos</codeph> that is less than or
+        equal to 15.
+      </p>
+
+<codeblock>insert into animals values ('California Condor', 'Bird', 15), ('Andean Condor', 'Bird', 15)
+
+select name, kind, cume_dist() over (order by kilos) from animals where kind = 'Bird';
++-------------------+------+-----------------------+
+| name              | kind | cume_dist() OVER(...) |
++-------------------+------+-----------------------+
+| Ostrich           | Bird | 1                     |
+| Condor            | Bird | 0.8                   |
+| California Condor | Bird | 0.8                   |
+| Andean Condor     | Bird | 0.8                   |
+| Owl               | Bird | 0.2                   |
++-------------------+------+-----------------------+
+</codeblock>
+
+      <p>
+        The following example shows how to use an <codeph>ORDER BY</codeph> clause in the outer block
+        to order the result set in case of ties. Here, all the <q>Bird</q> rows are together, then in descending order
+        by the result of the <codeph>CUME_DIST()</codeph> function, and all tied <codeph>CUME_DIST()</codeph>
+        values are ordered by the animal name.
+      </p>
+
+<codeblock>select name, kind, cume_dist() over (partition by kind order by kilos) as ordering
+  from animals
+where
+  kind = 'Bird'
+order by kind, ordering desc, name;
++-------------------+------+----------+
+| name              | kind | ordering |
++-------------------+------+----------+
+| Ostrich           | Bird | 1        |
+| Andean Condor     | Bird | 0.8      |
+| California Condor | Bird | 0.8      |
+| Condor            | Bird | 0.8      |
+| Owl               | Bird | 0.2      |
++-------------------+------+----------+
+</codeblock>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.0.0" id="dense_rank">
+
+    <title>DENSE_RANK Function</title>
+
+    <conbody>
+
+      <p>
+        Returns an ascending sequence of integers, starting with 1. The output sequence produces duplicate integers
+        for duplicate values of the <codeph>ORDER BY</codeph> expressions. After generating duplicate output values
+        for the <q>tied</q> input values, the function continues the sequence with the next higher integer.
+        Therefore, the sequence contains duplicates but no gaps when the input contains duplicates. Starts the
+        sequence over for each group produced by the <codeph>PARTITIONED BY</codeph> clause.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>DENSE_RANK() OVER([<varname>partition_by_clause</varname>] <varname>order_by_clause</varname>)</codeblock>
+
+      <p>
+        The <codeph>PARTITION BY</codeph> clause is optional. The <codeph>ORDER BY</codeph> clause is required. The
+        window clause is not allowed.
+      </p>
+
+<!-- Can make the text for ROW_NUMBER, RANK, and DENSE_RANK identical
+     so it can be conref'ed in all 3 places. -->
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p>
+        Often used for top-N and bottom-N queries. For example, it could produce a <q>top 10</q> report including
+        all the items with the 10 highest values, even if several items tied for 1st place.
+      </p>
+
+      <p>
+        Similar to <codeph>ROW_NUMBER</codeph> and <codeph>RANK</codeph>. These functions differ in how they treat
+        duplicate combinations of values.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/added_in_20"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following example demonstrates how the <codeph>DENSE_RANK()</codeph> function identifies where each
+        value <q>places</q> in the result set, producing the same result for duplicate values, but with a strict
+        sequence from 1 to the number of groups. For example, when results are ordered by the <codeph>X</codeph>
+        column, both <codeph>1</codeph> values are tied for first; both <codeph>2</codeph> values are tied for
+        second; and so on.
+      </p>
+
+<codeblock>select x, dense_rank() over(order by x) as rank, property from int_t;
++----+------+----------+
+| x  | rank | property |
++----+------+----------+
+| 1  | 1    | square   |
+| 1  | 1    | odd      |
+| 2  | 2    | even     |
+| 2  | 2    | prime    |
+| 3  | 3    | prime    |
+| 3  | 3    | odd      |
+| 4  | 4    | even     |
+| 4  | 4    | square   |
+| 5  | 5    | odd      |
+| 5  | 5    | prime    |
+| 6  | 6    | even     |
+| 6  | 6    | perfect  |
+| 7  | 7    | lucky    |
+| 7  | 7    | lucky    |
+| 7  | 7    | lucky    |
+| 7  | 7    | odd      |
+| 7  | 7    | prime    |
+| 8  | 8    | even     |
+| 9  | 9    | square   |
+| 9  | 9    | odd      |
+| 10 | 10   | round    |
+| 10 | 10   | even     |
++----+------+----------+
+</codeblock>
+
+      <p>
+        The following examples show how the <codeph>DENSE_RANK()</codeph> function is affected by the
+        <codeph>PARTITION</codeph> property within the <codeph>ORDER BY</codeph> clause.
+      </p>
+
+      <p>
+        Partitioning by the <codeph>PROPERTY</codeph> column groups all the even, odd, and so on values together,
+        and <codeph>DENSE_RANK()</codeph> returns the place of each value within the group, producing several
+        ascending sequences.
+      </p>
+
+<codeblock>select x, dense_rank() over(partition by property order by x) as rank, property from int_t;
++----+------+----------+
+| x  | rank | property |
++----+------+----------+
+| 2  | 1    | even     |
+| 4  | 2    | even     |
+| 6  | 3    | even     |
+| 8  | 4    | even     |
+| 10 | 5    | even     |
+| 7  | 1    | lucky    |
+| 7  | 1    | lucky    |
+| 7  | 1    | lucky    |
+| 1  | 1    | odd      |
+| 3  | 2    | odd      |
+| 5  | 3    | odd      |
+| 7  | 4    | odd      |
+| 9  | 5    | odd      |
+| 6  | 1    | perfect  |
+| 2  | 1    | prime    |
+| 3  | 2    | prime    |
+| 5  | 3    | prime    |
+| 7  | 4    | prime    |
+| 10 | 1    | round    |
+| 1  | 1    | square   |
+| 4  | 2    | square   |
+| 9  | 3    | square   |
++----+------+----------+
+</codeblock>
+
+      <p>
+        Partitioning by the <codeph>X</codeph> column groups all the duplicate numbers together and returns the
+        place each each value within the group; because each value occurs only 1 or 2 times,
+        <codeph>DENSE_RANK()</codeph> designates each <codeph>X</codeph> value as either first or second within its
+        group.
+      </p>
+
+<codeblock>select x, dense_rank() over(partition by x order by property) as rank, property from int_t;
++----+------+----------+
+| x  | rank | property |
++----+------+----------+
+| 1  | 1    | odd      |
+| 1  | 2    | square   |
+| 2  | 1    | even     |
+| 2  | 2    | prime    |
+| 3  | 1    | odd      |
+| 3  | 2    | prime    |
+| 4  | 1    | even     |
+| 4  | 2    | square   |
+| 5  | 1    | odd      |
+| 5  | 2    | prime    |
+| 6  | 1    | even     |
+| 6  | 2    | perfect  |
+| 7  | 1    | lucky    |
+| 7  | 1    | lucky    |
+| 7  | 1    | lucky    |
+| 7  | 2    | odd      |
+| 7  | 3    | prime    |
+| 8  | 1    | even     |
+| 9  | 1    | odd      |
+| 9  | 2    | square   |
+| 10 | 1    | even     |
+| 10 | 2    | round    |
++----+------+----------+
+</codeblock>
+
+      <p>
+        The following example shows how <codeph>DENSE_RANK()</codeph> produces a continuous sequence while still
+        allowing for ties. In this case, Croesus and Midas both have the second largest fortune, while Crassus has
+        the third largest. (In <xref href="impala_analytic_functions.xml#rank"/>, you see a similar query with the
+        <codeph>RANK()</codeph> function that shows that while Crassus has the third largest fortune, he is the
+        fourth richest person.)
+      </p>
+
+<codeblock>select dense_rank() over (order by net_worth desc) as placement, name, net_worth from wealth order by placement, name;
++-----------+---------+---------------+
+| placement | name    | net_worth     |
++-----------+---------+---------------+
+| 1         | Solomon | 2000000000.00 |
+| 2         | Croesus | 1000000000.00 |
+| 2         | Midas   | 1000000000.00 |
+| 3         | Crassus | 500000000.00  |
+| 4         | Scrooge | 80000000.00   |
++-----------+---------+---------------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_analytic_functions.xml#rank"/>, <xref href="impala_analytic_functions.xml#row_number"/>
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.0.0" id="first_value">
+
+    <title>FIRST_VALUE Function</title>
+
+    <conbody>
+
+      <p>
+        Returns the expression value from the first row in the window. The return value is <codeph>NULL</codeph> if
+        the input expression is <codeph>NULL</codeph>.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>FIRST_VALUE(<varname>expr</varname>) OVER([<varname>partition_by_clause</varname>] <varname>order_by_clause</varname> [<varname>window_clause</varname>])</codeblock>
+
+      <p>
+        The <codeph>PARTITION BY</codeph> clause is optional. The <codeph>ORDER BY</codeph> clause is required. The
+        window clause is optional.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p>
+        If any duplicate values occur in the tuples evaluated by the <codeph>ORDER BY</codeph> clause, the result
+        of this function is not deterministic. Consider adding additional <codeph>ORDER BY</codeph> columns to
+        ensure consistent ordering.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/added_in_20"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following example shows a table with a wide variety of country-appropriate greetings. For consistency,
+        we want to standardize on a single greeting for each country. The <codeph>FIRST_VALUE()</codeph> function
+        helps to produce a mail merge report where every person from the same country is addressed with the same
+        greeting.
+      </p>
+
+<codeblock>select name, country, greeting from mail_merge
++---------+---------+--------------+
+| name    | country | greeting     |
++---------+---------+--------------+
+| Pete    | USA     | Hello        |
+| John    | USA     | Hi           |
+| Boris   | Germany | Guten tag    |
+| Michael | Germany | Guten morgen |
+| Bjorn   | Sweden  | Hej          |
+| Mats    | Sweden  | Tja          |
++---------+---------+--------------+
+
+select country, name,
+  first_value(greeting)
+    over (partition by country order by name, greeting) as greeting
+  from mail_merge;
++---------+---------+-----------+
+| country | name    | greeting  |
++---------+---------+-----------+
+| Germany | Boris   | Guten tag |
+| Germany | Michael | Guten tag |
+| Sweden  | Bjorn   | Hej       |
+| Sweden  | Mats    | Hej       |
+| USA     | John    | Hi        |
+| USA     | Pete    | Hi        |
++---------+---------+-----------+
+</codeblock>
+
+      <p>
+        Changing the order in which the names are evaluated changes which greeting is applied to each group.
+      </p>
+
+<codeblock>select country, name,
+  first_value(greeting)
+    over (partition by country order by name desc, greeting) as greeting
+  from mail_merge;
++---------+---------+--------------+
+| country | name    | greeting     |
++---------+---------+--------------+
+| Germany | Michael | Guten morgen |
+| Germany | Boris   | Guten morgen |
+| Sweden  | Mats    | Tja          |
+| Sweden  | Bjorn   | Tja          |
+| USA     | Pete    | Hello        |
+| USA     | John    | Hello        |
++---------+---------+--------------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_analytic_functions.xml#last_value"/>
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.0.0" id="lag">
+
+    <title>LAG Function</title>
+
+    <conbody>
+
+      <p>
+        This function returns the value of an expression using column values from a preceding row. You specify an
+        integer offset, which designates a row position some number of rows previous to the current row. Any column
+        references in the expression argument refer to column values from that prior row. Typically, the table
+        contains a time sequence or numeric sequence column that clearly distinguishes the ordering of the rows.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>LAG (<varname>expr</varname> [, <varname>offset</varname>] [, <varname>default</varname>])
+  OVER ([<varname>partition_by_clause</varname>] <varname>order_by_clause</varname>)</codeblock>
+
+      <p>
+        The <codeph>ORDER BY</codeph> clause is required. The <codeph>PARTITION BY</codeph> clause is optional. The
+        window clause is not allowed.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p>
+        Sometimes used an an alternative to doing a self-join.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/added_in_20"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following example uses the same stock data created in <xref href="#window_clause"/>. For each day, the
+        query prints the closing price alongside the previous day's closing price. The first row for each stock
+        symbol has no previous row, so that <codeph>LAG()</codeph> value is <codeph>NULL</codeph>.
+      </p>
+
+<codeblock>select stock_symbol, closing_date, closing_price,
+    lag(closing_price,1) over (partition by stock_symbol order by closing_date) as "yesterday closing"
+  from stock_ticker
+    order by closing_date;
++--------------+---------------------+---------------+-------------------+
+| stock_symbol | closing_date        | closing_price | yesterday closing |
++--------------+---------------------+---------------+-------------------+
+| JDR          | 2014-09-13 00:00:00 | 12.86         | NULL              |
+| JDR          | 2014-09-14 00:00:00 | 12.89         | 12.86             |
+| JDR          | 2014-09-15 00:00:00 | 12.94         | 12.89             |
+| JDR          | 2014-09-16 00:00:00 | 12.55         | 12.94             |
+| JDR          | 2014-09-17 00:00:00 | 14.03         | 12.55             |
+| JDR          | 2014-09-18 00:00:00 | 14.75         | 14.03             |
+| JDR          | 2014-09-19 00:00:00 | 13.98         | 14.75             |
++--------------+---------------------+---------------+-------------------+
+</codeblock>
+
+      <p>
+        The following example does an arithmetic operation between the current row and a value from the previous
+        row, to produce a delta value for each day. This example also demonstrates how <codeph>ORDER BY</codeph>
+        works independently in the different parts of the query. The <codeph>ORDER BY closing_date</codeph> in the
+        <codeph>OVER</codeph> clause makes the query analyze the rows in chronological order. Then the outer query
+        block uses <codeph>ORDER BY closing_date DESC</codeph> to present the results with the most recent date
+        first.
+      </p>
+
+<codeblock>select stock_symbol, closing_date, closing_price,
+    cast(
+      closing_price - lag(closing_price,1) over
+        (partition by stock_symbol order by closing_date)
+      as decimal(8,2)
+    )
+    as "change from yesterday"
+  from stock_ticker
+    order by closing_date desc;
++--------------+---------------------+---------------+-----------------------+
+| stock_symbol | closing_date        | closing_price | change from yesterday |
++--------------+---------------------+---------------+-----------------------+
+| JDR          | 2014-09-19 00:00:00 | 13.98         | -0.76                 |
+| JDR          | 2014-09-18 00:00:00 | 14.75         | 0.72                  |
+| JDR          | 2014-09-17 00:00:00 | 14.03         | 1.47                  |
+| JDR          | 2014-09-16 00:00:00 | 12.55         | -0.38                 |
+| JDR          | 2014-09-15 00:00:00 | 12.94         | 0.04                  |
+| JDR          | 2014-09-14 00:00:00 | 12.89         | 0.03                  |
+| JDR          | 2014-09-13 00:00:00 | 12.86         | NULL                  |
++--------------+---------------------+---------------+-----------------------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        This function is the converse of <xref href="impala_analytic_functions.xml#lead"/>.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.0.0" id="last_value">
+
+    <title>LAST_VALUE Function</title>
+
+    <conbody>
+
+      <p>
+        Returns the expression value from the last row in the window. This same value is repeated for all result
+        rows for the group. The return value is <codeph>NULL</codeph> if the input expression is
+        <codeph>NULL</codeph>.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>LAST_VALUE(<varname>expr</varname>) OVER([<varname>partition_by_clause</varname>] <varname>order_by_clause</varname> [<varname>window_clause</varname>])</codeblock>
+
+      <p>
+        The <codeph>PARTITION BY</codeph> clause is optional. The <codeph>ORDER BY</codeph> clause is required. The
+        window clause is optional.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p>
+        If any duplicate values occur in the tuples evaluated by the <codeph>ORDER BY</codeph> clause, the result
+        of this function is not deterministic. Consider adding additional <codeph>ORDER BY</codeph> columns to
+        ensure consistent ordering.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/added_in_20"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following example uses the same <codeph>MAIL_MERGE</codeph> table as in the example for
+        <xref href="impala_analytic_functions.xml#first_value"/>. Because the default window when <codeph>ORDER
+        BY</codeph> is used is <codeph>BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW</codeph>, the query requires the
+        <codeph>UNBOUNDED FOLLOWING</codeph> to look ahead to subsequent rows and find the last value for each
+        country.
+      </p>
+
+<codeblock>select country, name,
+  last_value(greeting) over (
+    partition by country order by name, greeting
+    rows between unbounded preceding and unbounded following
+  ) as greeting
+  from mail_merge
++---------+---------+--------------+
+| country | name    | greeting     |
++---------+---------+--------------+
+| Germany | Boris   | Guten morgen |
+| Germany | Michael | Guten morgen |
+| Sweden  | Bjorn   | Tja          |
+| Sweden  | Mats    | Tja          |
+| USA     | John    | Hello        |
+| USA     | Pete    | Hello        |
++---------+---------+--------------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_analytic_functions.xml#first_value"/>
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.0.0" id="lead">
+
+    <title>LEAD Function</title>
+
+    <conbody>
+
+      <p>
+        This function returns the value of an expression using column values from a following row. You specify an
+        integer offset, which designates a row position some number of rows after to the current row. Any column
+        references in the expression argument refer to column values from that later row. Typically, the table
+        contains a time sequence or numeric sequence column that clearly distinguishes the ordering of the rows.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>LEAD (<varname>expr</varname> [, <varname>offset</varname>] [, <varname>default</varname>])
+  OVER ([<varname>partition_by_clause</varname>] <varname>order_by_clause</varname>)</codeblock>
+
+      <p>
+        The <codeph>ORDER BY</codeph> clause is required. The <codeph>PARTITION BY</codeph> clause is optional. The
+        window clause is not allowed.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p>
+        Sometimes used an an alternative to doing a self-join.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/added_in_20"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following example uses the same stock data created in <xref href="#window_clause"/>. The query analyzes
+        the closing price for a stock symbol, and for each day evaluates if the closing price for the following day
+        is higher or lower.
+      </p>
+
+<codeblock>select stock_symbol, closing_date, closing_price,
+  case
+    (lead(closing_price,1)
+      over (partition by stock_symbol order by closing_date)
+        - closing_price) &gt; 0
+    when true then "higher"
+    when false then "flat or lower"
+  end as "trending"
+from stock_ticker
+  order by closing_date;
++--------------+---------------------+---------------+---------------+
+| stock_symbol | closing_date        | closing_price | trending      |
++--------------+---------------------+---------------+---------------+
+| JDR          | 2014-09-13 00:00:00 | 12.86         | higher        |
+| JDR          | 2014-09-14 00:00:00 | 12.89         | higher        |
+| JDR          | 2014-09-15 00:00:00 | 12.94         | flat or lower |
+| JDR          | 2014-09-16 00:00:00 | 12.55         | higher        |
+| JDR          | 2014-09-17 00:00:00 | 14.03         | higher        |
+| JDR          | 2014-09-18 00:00:00 | 14.75         | flat or lower |
+| JDR          | 2014-09-19 00:00:00 | 13.98         | NULL          |
++--------------+---------------------+---------------+---------------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        This function is the converse of <xref href="impala_analytic_functions.xml#lag"/>.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="max_analytic">
+
+    <title>MAX Function - Analytic Context</title>
+
+    <conbody>
+
+      <p>
+        You can include an <codeph>OVER</codeph> clause with a call to this function to use it as an analytic
+        function. See <xref href="impala_max.xml#max"/> for details and examples.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="min_analytic">
+
+    <title>MIN Function - Analytic Context</title>
+
+    <conbody>
+
+      <p>
+        You can include an <codeph>OVER</codeph> clause with a call to this function to use it as an analytic
+        function. See <xref href="impala_min.xml#min"/> for details and examples.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept audience="Cloudera" rev="2.x.x" id="nth_value">
+
+    <title>NTH_VALUE Function</title>
+
+    <conbody>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.3.0" id="ntile">
+
+    <title>NTILE Function (<keyword keyref="impala23"/> or higher only)</title>
+
+    <conbody>
+
+      <p>
+        Returns the <q>bucket number</q> associated with each row, between 1 and the value of an expression. For
+        example, creating 100 buckets puts the lowest 1% of values in the first bucket, while creating 10 buckets
+        puts the lowest 10% of values in the first bucket. Each partition can have a different number of buckets.
+<!-- What's the syntax or data distribution that would create a different number of buckets per partition? -->
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>NTILE (<varname>expr</varname> [, <varname>offset</varname> ...]
+  OVER ([<varname>partition_by_clause</varname>] <varname>order_by_clause</varname>)</codeblock>
+
+      <p>
+        The <codeph>ORDER BY</codeph> clause is required. The <codeph>PARTITION BY</codeph> clause is optional. The
+        window clause is not allowed.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p>
+        The <q>ntile</q> name is derived from the practice of dividing result sets into fourths (quartile), tenths
+        (decile), and so on. The <codeph>NTILE()</codeph> function divides the result set based on an arbitrary
+        percentile value.
+      </p>
+
+      <p>
+        The number of buckets must be a positive integer.
+      </p>
+
+      <p>
+        The number of items in each bucket is identical or almost so, varying by at most 1. If the number of items
+        does not divide evenly between the buckets, the remaining N items are divided evenly among the first N
+        buckets.
+      </p>
+
+      <p>
+        If the number of buckets N is greater than the number of input rows in the partition, then the first N
+        buckets each contain one item, and the remaining buckets are empty.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following example shows divides groups of animals into 4 buckets based on their weight. The
+        <codeph>ORDER BY ... DESC</codeph> clause in the <codeph>OVER()</codeph> clause means that the heaviest 25%
+        are in the first group, and the lightest 25% are in the fourth group. (The <codeph>ORDER BY</codeph> in the
+        outermost part of the query shows how you can order the final result set independently from the order in
+        which the rows are evaluated by the <codeph>OVER()</codeph> clause.) Because there are 9 rows in the group,
+        divided into 4 buckets, the first bucket receives the extra item.
+      </p>
+
+<codeblock>create table animals (name string, kind string, kilos decimal(9,3));
+
+insert into animals values
+  ('Elephant', 'Mammal', 4000), ('Giraffe', 'Mammal', 1200), ('Mouse', 'Mammal', 0.020),
+  ('Condor', 'Bird', 15), ('Horse', 'Mammal', 500), ('Owl', 'Bird', 2.5),
+  ('Ostrich', 'Bird', 145), ('Polar bear', 'Mammal', 700), ('Housecat', 'Mammal', 5);
+
+select name, ntile(4) over (order by kilos desc) as quarter
+  from animals
+order by quarter desc;
++------------+---------+
+| name       | quarter |
++------------+---------+
+| Owl        | 4       |
+| Mouse      | 4       |
+| Condor     | 3       |
+| Housecat   | 3       |
+| Horse      | 2       |
+| Ostrich    | 2       |
+| Elephant   | 1       |
+| Giraffe    | 1       |
+| Polar bear | 1       |
++------------+---------+
+</codeblock>
+
+      <p>
+        The following examples show how the <codeph>PARTITION</codeph> clause works for the
+        <codeph>NTILE()</codeph> function. Here, we divide each kind of animal (mammal or bird) into 2 buckets,
+        the heavier half and the lighter half.
+      </p>
+
+<codeblock>select name, kind, ntile(2) over (partition by kind order by kilos desc) as half
+  from animals
+order by kind;
++------------+--------+------+
+| name       | kind   | half |
++------------+--------+------+
+| Ostrich    | Bird   | 1    |
+| Condor     | Bird   | 1    |
+| Owl        | Bird   | 2    |
+| Elephant   | Mammal | 1    |
+| Giraffe    | Mammal | 1    |
+| Polar bear | Mammal | 1    |
+| Horse      | Mammal | 2    |
+| Housecat   | Mammal | 2    |
+| Mouse      | Mammal | 2    |
++------------+--------+------+
+</codeblock>
+
+      <p>
+        Again, the result set can be ordered independently
+        from the analytic evaluation. This next example lists all the animals heaviest to lightest,
+        showing that elephant and giraffe are in the <q>top half</q> of mammals by weight, while
+        housecat and mouse are in the <q>bottom half</q>.
+      </p>
+
+<codeblock>select name, kind, ntile(2) over (partition by kind order by kilos desc) as half
+  from animals
+order by kilos desc;
++------------+--------+------+
+| name       | kind   | half |
++------------+--------+------+
+| Elephant   | Mammal | 1    |
+| Giraffe    | Mammal | 1    |
+| Polar bear | Mammal | 1    |
+| Horse      | Mammal | 2    |
+| Ostrich    | Bird   | 1    |
+| Condor     | Bird   | 1    |
+| Housecat   | Mammal | 2    |
+| Owl        | Bird   | 2    |
+| Mouse      | Mammal | 2    |
++------------+--------+------+
+</codeblock>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.3.0" id="percent_rank">
+
+    <title>PERCENT_RANK Function (<keyword keyref="impala23"/> or higher only)</title>
+
+    <conbody>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>PERCENT_RANK (<varname>expr</varname>)
+  OVER ([<varname>partition_by_clause</varname>] <varname>order_by_clause</varname>)
+</codeblock>
+
+      <p>
+      Calculates the rank, expressed as a percentage, of each row within a group of rows.
+      If <codeph>rank</codeph> is the value for that same row from the <codeph>RANK()</codeph> function (from 1 to the total number of rows in the partition group),
+      then the <codeph>PERCENT_RANK()</codeph> value is calculated as <codeph>(<varname>rank</varname> - 1) / (<varname>rows_in_group</varname> - 1)</codeph> .
+      If there is only a single item in the partition group, its <codeph>PERCENT_RANK()</codeph> value is 0.
+      </p>
+
+      <p>
+        The <codeph>ORDER BY</codeph> clause is required. The <codeph>PARTITION BY</codeph> clause is optional. The
+        window clause is not allowed.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p>
+        This function is similar to the <codeph>RANK</codeph> and <codeph>CUME_DIST()</codeph> functions: it returns an ascending sequence representing the position of each
+        row within the rows of the same partition group. The actual numeric sequence is calculated differently,
+        and the handling of duplicate (tied) values is different.
+      </p>
+
+      <p>
+        The return values range from 0 to 1 inclusive.
+        The first row in each partition group always has the value 0.
+        A <codeph>NULL</codeph> value is considered the lowest possible value.
+        In the case of duplicate input values, all the corresponding rows in the result set
+        have an identical value: the lowest <codeph>PERCENT_RANK()</codeph> value of those
+        tied rows. (In contrast to <codeph>CUME_DIST()</codeph>, where all tied rows have
+        the highest <codeph>CUME_DIST()</codeph> value.)
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following example uses the same <codeph>ANIMALS</codeph> table as the examples for <codeph>CUME_DIST()</codeph>
+        and <codeph>NTILE()</codeph>, with a few additional rows to illustrate the results where some values are
+        <codeph>NULL</codeph> or there is only a single row in a partition group.
+      </p>
+
+<codeblock>insert into animals values ('Komodo dragon', 'Reptile', 70);
+insert into animals values ('Unicorn', 'Mythical', NULL);
+insert into animals values ('Fire-breathing dragon', 'Mythical', NULL);
+</codeblock>
+
+      <p>
+        As with <codeph>CUME_DIST()</codeph>, there is an ascending sequence for each kind of animal.
+        For example, the <q>Birds</q> and <q>Mammals</q> rows each have a <codeph>PERCENT_RANK()</codeph> sequence
+        that ranges from 0 to 1.
+        The <q>Reptile</q> row has a <codeph>PERCENT_RANK()</codeph> of 0 because that partition group contains only a single item.
+        Both <q>Mythical</q> animals have a <codeph>PERCENT_RANK()</codeph> of 0 because
+        a <codeph>NULL</codeph> is considered the lowest value within its partition group.
+      </p>
+
+<codeblock>select name, kind, percent_rank() over (partition by kind order by kilos) from animals;
++-----------------------+----------+--------------------------+
+| name                  | kind     | percent_rank() OVER(...) |
++-----------------------+----------+--------------------------+
+| Mouse                 | Mammal   | 0                        |
+| Housecat              | Mammal   | 0.2                      |
+| Horse                 | Mammal   | 0.4                      |
+| Polar bear            | Mammal   | 0.6                      |
+| Giraffe               | Mammal   | 0.8                      |
+| Elephant              | Mammal   | 1                        |
+| Komodo dragon         | Reptile  | 0                        |
+| Owl                   | Bird     | 0                        |
+| California Condor     | Bird     | 0.25                     |
+| Andean Condor         | Bird     | 0.25                     |
+| Condor                | Bird     | 0.25                     |
+| Ostrich               | Bird     | 1                        |
+| Fire-breathing dragon | Mythical | 0                        |
+| Unicorn               | Mythical | 0                        |
++-----------------------+----------+--------------------------+
+</codeblock>
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.0.0" id="rank">
+
+    <title>RANK Function</title>
+
+    <conbody>
+
+      <p>
+        Returns an ascending sequence of integers, starting with 1. The output sequence produces duplicate integers
+        for duplicate values of the <codeph>ORDER BY</codeph> expressions. After generating duplicate output values
+        for the <q>tied</q> input values, the function increments the sequence by the number of tied values.
+        Therefore, the sequence contains both duplicates and gaps when the input contains duplicates. Starts the
+        sequence over for each group produced by the <codeph>PARTITIONED BY</codeph> clause.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>RANK() OVER([<varname>partition_by_clause</varname>] <varname>order_by_clause</varname>)</codeblock>
+
+      <p>
+        The <codeph>PARTITION BY</codeph> clause is optional. The <codeph>ORDER BY</codeph> clause is required. The
+        window clause is not allowed.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+<!-- Make a little tutorial to show these 3 functions side-by-side and illustrate their difference. -->
+
+      <p>
+        Often used for top-N and bottom-N queries. For example, it could produce a <q>top 10</q> report including
+        several items that were tied for 10th place.
+      </p>
+
+      <p>
+        Similar to <codeph>ROW_NUMBER</codeph> and <codeph>DENSE_RANK</codeph>. These functions differ in how they
+        treat duplicate combinations of values.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/added_in_20"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following example demonstrates how the <codeph>RANK()</codeph> function identifies where each value
+        <q>places</q> in the result set, producing the same result for duplicate values, and skipping values in the
+        sequence to account for the number of duplicates. For example, when results are ordered by the
+        <codeph>X</codeph> column, both <codeph>1</codeph> values are tied for first; both <codeph>2</codeph>
+        values are tied for third; and so on.
+      </p>
+
+<codeblock>select x, rank() over(order by x) as rank, property from int_t;
++----+------+----------+
+| x  | rank | property |
++----+------+----------+
+| 1  | 1    | square   |
+| 1  | 1    | odd      |
+| 2  | 3    | even     |
+| 2  | 3    | prime    |
+| 3  | 5    | prime    |
+| 3  | 5    | odd      |
+| 4  | 7    | even     |
+| 4  | 7    | square   |
+| 5  | 9    | odd      |
+| 5  | 9    | prime    |
+| 6  | 11   | even     |
+| 6  | 11   | perfect  |
+| 7  | 13   | lucky    |
+| 7  | 13   | lucky    |
+| 7  | 13   | lucky    |
+| 7  | 13   | odd      |
+| 7  | 13   | prime    |
+| 8  | 18   | even     |
+| 9  | 19   | square   |
+| 9  | 19   | odd      |
+| 10 | 21   | round    |
+| 10 | 21   | even     |
++----+------+----------+
+</codeblock>
+
+      <p>
+        The following examples show how the <codeph>RANK()</codeph> function is affected by the
+        <codeph>PARTITION</codeph> property within the <codeph>ORDER BY</codeph> clause.
+      </p>
+
+      <p>
+        Partitioning by the <codeph>PROPERTY</codeph> column groups all the even, odd, and so on values together,
+        and <codeph>RANK()</codeph> returns the place of each value within the group, producing several ascending
+        sequences.
+      </p>
+
+<codeblock>select x, rank() over(partition by property order by x) as rank, property from int_t;
++----+------+----------+
+| x  | rank | property |
++----+------+----------+
+| 2  | 1    | even     |
+| 4  | 2    | even     |
+| 6  | 3    | even     |
+| 8  | 4    | even     |
+| 10 | 5    | even     |
+| 7  | 1    | lucky    |
+| 7  | 1    | lucky    |
+| 7  | 1    | lucky    |
+| 1  | 1    | odd      |
+| 3  | 2    | odd      |
+| 5  | 3    | odd      |
+| 7  | 4    | odd      |
+| 9  | 5    | odd      |
+| 6  | 1    | perfect  |
+| 2  | 1    | prime    |
+| 3  | 2    | prime    |
+| 5  | 3    | prime    |
+| 7  | 4    | prime    |
+| 10 | 1    | round    |
+| 1  | 1    | square   |
+| 4  | 2    | square   |
+| 9  | 3    | square   |
++----+------+----------+
+</codeblock>
+
+      <p>
+        Partitioning by the <codeph>X</codeph> column groups all the duplicate numbers together and returns the
+        place each each value within the group; because each value occurs only 1 or 2 times,
+        <codeph>RANK()</codeph> designates each <codeph>X</codeph> value as either first or second within its
+        group.
+      </p>
+
+<codeblock>select x, rank() over(partition by x order by property) as rank, property from int_t;
++----+------+----------+
+| x  | rank | property |
++----+------+----------+
+| 1  | 1    | odd      |
+| 1  | 2    | square   |
+| 2  | 1    | even     |
+| 2  | 2    | prime    |
+| 3  | 1    | odd      |
+| 3  | 2    | prime    |
+| 4  | 1    | even     |
+| 4  | 2    | square   |
+| 5  | 1    | odd      |
+| 5  | 2    | prime    |
+| 6  | 1    | even     |
+| 6  | 2    | perfect  |
+| 7  | 1    | lucky    |
+| 7  | 1    | lucky    |
+| 7  | 1    | lucky    |
+| 7  | 4    | odd      |
+| 7  | 5    | prime    |
+| 8  | 1    | even     |
+| 9  | 1    | odd      |
+| 9  | 2    | square   |
+| 10 | 1    | even     |
+| 10 | 2    | round    |
++----+------+----------+
+</codeblock>
+
+      <p>
+        The following example shows how a magazine might prepare a list of history's wealthiest people. Croesus and
+        Midas are tied for second, then Crassus is fourth.
+      </p>
+
+<codeblock>select rank() over (order by net_worth desc) as rank, name, net_worth from wealth order by rank, name;
++------+---------+---------------+
+| rank | name    | net_worth     |
++------+---------+---------------+
+| 1    | Solomon | 2000000000.00 |
+| 2    | Croesus | 1000000000.00 |
+| 2    | Midas   | 1000000000.00 |
+| 4    | Crassus | 500000000.00  |
+| 5    | Scrooge | 80000000.00   |
++------+---------+---------------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_analytic_functions.xml#dense_rank"/>,
+        <xref href="impala_analytic_functions.xml#row_number"/>
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.0.0" id="row_number">
+
+    <title>ROW_NUMBER Function</title>
+
+    <conbody>
+
+      <p>
+        Returns an ascending sequence of integers, starting with 1. Starts the sequence over for each group
+        produced by the <codeph>PARTITIONED BY</codeph> clause. The output sequence includes different values for
+        duplicate input values. Therefore, the sequence never contains any duplicates or gaps, regardless of
+        duplicate input values.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>ROW_NUMBER() OVER([<varname>partition_by_clause</varname>] <varname>order_by_clause</varname>)</codeblock>
+
+      <p>
+        The <codeph>ORDER BY</codeph> clause is required. The <codeph>PARTITION BY</codeph> clause is optional. The
+        window clause is not allowed.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+      <p>
+        Often used for top-N and bottom-N queries where the input values are known to be unique, or precisely N
+        rows are needed regardless of duplicate values.
+      </p>
+
+      <p>
+        Because its result value is different for each row in the result set (when used without a <codeph>PARTITION
+        BY</codeph> clause), <codeph>ROW_NUMBER()</codeph> can be used to synthesize unique numeric ID values, for
+        example for result sets involving unique values or tuples.
+      </p>
+
+      <p>
+        Similar to <codeph>RANK</codeph> and <codeph>DENSE_RANK</codeph>. These functions differ in how they treat
+        duplicate combinations of values.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/added_in_20"/>
+
+      <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+      <p>
+        The following example demonstrates how <codeph>ROW_NUMBER()</codeph> produces a continuous numeric
+        sequence, even though some values of <codeph>X</codeph> are repeated.
+      </p>
+
+<codeblock>select x, row_number() over(order by x, property) as row_number, property from int_t;
++----+------------+----------+
+| x  | row_number | property |
++----+------------+----------+
+| 1  | 1          | odd      |
+| 1  | 2          | square   |
+| 2  | 3          | even     |
+| 2  | 4          | prime    |
+| 3  | 5          | odd      |
+| 3  | 6          | prime    |
+| 4  | 7          | even     |
+| 4  | 8          | square   |
+| 5  | 9          | odd      |
+| 5  | 10         | prime    |
+| 6  | 11         | even     |
+| 6  | 12         | perfect  |
+| 7  | 13         | lucky    |
+| 7  | 14         | lucky    |
+| 7  | 15         | lucky    |
+| 7  | 16         | odd      |
+| 7  | 17         | prime    |
+| 8  | 18         | even     |
+| 9  | 19         | odd      |
+| 9  | 20         | square   |
+| 10 | 21         | even     |
+| 10 | 22         | round    |
++----+------------+----------+
+</codeblock>
+
+      <p>
+        The following example shows how a financial institution might assign customer IDs to some of history's
+        wealthiest figures. Although two of the people have identical net worth figures, unique IDs are required
+        for this purpose. <codeph>ROW_NUMBER()</codeph> produces a sequence of five different values for the five
+        input rows.
+      </p>
+
+<codeblock>select row_number() over (order by net_worth desc) as account_id, name, net_worth
+  from wealth order by account_id, name;
++------------+---------+---------------+
+| account_id | name    | net_worth     |
++------------+---------+---------------+
+| 1          | Solomon | 2000000000.00 |
+| 2          | Croesus | 1000000000.00 |
+| 3          | Midas   | 1000000000.00 |
+| 4          | Crassus | 500000000.00  |
+| 5          | Scrooge | 80000000.00   |
++------------+---------+---------------+
+</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/related_info"/>
+
+      <p>
+        <xref href="impala_analytic_functions.xml#rank"/>, <xref href="impala_analytic_functions.xml#dense_rank"/>
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="sum_analytic">
+
+    <title>SUM Function - Analytic Context</title>
+
+    <conbody>
+
+      <p>
+        You can include an <codeph>OVER</codeph> clause with a call to this function to use it as an analytic
+        function. See <xref href="impala_sum.xml#sum"/> for details and examples.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_appx_count_distinct.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_appx_count_distinct.xml b/docs/topics/impala_appx_count_distinct.xml
new file mode 100644
index 0000000..04078db
--- /dev/null
+++ b/docs/topics/impala_appx_count_distinct.xml
@@ -0,0 +1,81 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="2.0.0" id="appx_count_distinct">
+
+  <title>APPX_COUNT_DISTINCT Query Option (<keyword keyref="impala20"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>APPX_COUNT_DISTINCT</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Aggregate Functions"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.0.0">
+      <indexterm audience="Cloudera">APPX_COUNT_DISTINCT query option</indexterm>
+      Allows multiple <codeph>COUNT(DISTINCT)</codeph> operations within a single query, by internally rewriting
+      each <codeph>COUNT(DISTINCT)</codeph> to use the <codeph>NDV()</codeph> function. The resulting count is
+      approximate rather than precise.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_boolean"/>
+
+    <p conref="../shared/impala_common.xml#common/default_false_0"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      The following examples show how the <codeph>APPX_COUNT_DISTINCT</codeph> lets you work around the restriction
+      where a query can only evaluate <codeph>COUNT(DISTINCT <varname>col_name</varname>)</codeph> for a single
+      column. By default, you can count the distinct values of one column or another, but not both in a single
+      query:
+    </p>
+
+<codeblock>[localhost:21000] &gt; select count(distinct x) from int_t;
++-------------------+
+| count(distinct x) |
++-------------------+
+| 10                |
++-------------------+
+[localhost:21000] &gt; select count(distinct property) from int_t;
++--------------------------+
+| count(distinct property) |
++--------------------------+
+| 7                        |
++--------------------------+
+[localhost:21000] &gt; select count(distinct x), count(distinct property) from int_t;
+ERROR: AnalysisException: all DISTINCT aggregate functions need to have the same set of parameters
+as count(DISTINCT x); deviating function: count(DISTINCT property)
+</codeblock>
+
+    <p>
+      When you enable the <codeph>APPX_COUNT_DISTINCT</codeph> query option, now the query with multiple
+      <codeph>COUNT(DISTINCT)</codeph> works. The reason this behavior requires a query option is that each
+      <codeph>COUNT(DISTINCT)</codeph> is rewritten internally to use the <codeph>NDV()</codeph> function instead,
+      which provides an approximate result rather than a precise count.
+    </p>
+
+<codeblock>[localhost:21000] &gt; set APPX_COUNT_DISTINCT=true;
+[localhost:21000] &gt; select count(distinct x), count(distinct property) from int_t;
++-------------------+--------------------------+
+| count(distinct x) | count(distinct property) |
++-------------------+--------------------------+
+| 10                | 7                        |
++-------------------+--------------------------+
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_count.xml#count"/>,
+      <xref href="impala_distinct.xml#distinct"/>,
+      <xref href="impala_ndv.xml#ndv"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_appx_median.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_appx_median.xml b/docs/topics/impala_appx_median.xml
new file mode 100644
index 0000000..b97f6e3
--- /dev/null
+++ b/docs/topics/impala_appx_median.xml
@@ -0,0 +1,124 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.2.1" id="appx_median">
+
+  <title>APPX_MEDIAN Function</title>
+  <titlealts audience="PDF"><navtitle>APPX_MEDIAN</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="Aggregate Functions"/>
+      <data name="Category" value="Querying"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">appx_median() function</indexterm>
+      An aggregate function that returns a value that is approximately the median (midpoint) of values in the set
+      of input values.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>APPX_MEDIAN([DISTINCT | ALL] <varname>expression</varname>)
+</codeblock>
+
+    <p>
+      This function works with any input type, because the only requirement is that the type supports less-than and
+      greater-than comparison operators.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      Because the return value represents the estimated midpoint, it might not reflect the precise midpoint value,
+      especially if the cardinality of the input values is very high. If the cardinality is low (up to
+      approximately 20,000), the result is more accurate because the sampling considers all or almost all of the
+      different values.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/return_type_same_except_string"/>
+
+    <p>
+      The return value is always the same as one of the input values, not an <q>in-between</q> value produced by
+      averaging.
+    </p>
+
+<!-- <p conref="../shared/impala_common.xml#common/restrictions_sliding_window"/> -->
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/analytic_not_allowed_caveat"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      The following example uses a table of a million random floating-point numbers ranging up to approximately
+      50,000. The average is approximately 25,000. Because of the random distribution, we would expect the median
+      to be close to this same number. Computing the precise median is a more intensive operation than computing
+      the average, because it requires keeping track of every distinct value and how many times each occurs. The
+      <codeph>APPX_MEDIAN()</codeph> function uses a sampling algorithm to return an approximate result, which in
+      this case is close to the expected value. To make sure that the value is not substantially out of range due
+      to a skewed distribution, subsequent queries confirm that there are approximately 500,000 values higher than
+      the <codeph>APPX_MEDIAN()</codeph> value, and approximately 500,000 values lower than the
+      <codeph>APPX_MEDIAN()</codeph> value.
+    </p>
+
+<codeblock>[localhost:21000] &gt; select min(x), max(x), avg(x) from million_numbers;
++-------------------+-------------------+-------------------+
+| min(x)            | max(x)            | avg(x)            |
++-------------------+-------------------+-------------------+
+| 4.725693727250069 | 49994.56852674231 | 24945.38563793553 |
++-------------------+-------------------+-------------------+
+[localhost:21000] &gt; select appx_median(x) from million_numbers;
++----------------+
+| appx_median(x) |
++----------------+
+| 24721.6        |
++----------------+
+[localhost:21000] &gt; select count(x) as higher from million_numbers where x &gt; (select appx_median(x) from million_numbers);
++--------+
+| higher |
++--------+
+| 502013 |
++--------+
+[localhost:21000] &gt; select count(x) as lower from million_numbers where x &lt; (select appx_median(x) from million_numbers);
++--------+
+| lower  |
++--------+
+| 497987 |
++--------+
+</codeblock>
+
+    <p>
+      The following example computes the approximate median using a subset of the values from the table, and then
+      confirms that the result is a reasonable estimate for the midpoint.
+    </p>
+
+<codeblock>[localhost:21000] &gt; select appx_median(x) from million_numbers where x between 1000 and 5000;
++-------------------+
+| appx_median(x)    |
++-------------------+
+| 3013.107787358159 |
++-------------------+
+[localhost:21000] &gt; select count(x) as higher from million_numbers where x between 1000 and 5000 and x &gt; 3013.107787358159;
++--------+
+| higher |
++--------+
+| 37692  |
++--------+
+[localhost:21000] &gt; select count(x) as lower from million_numbers where x between 1000 and 5000 and x &lt; 3013.107787358159;
++-------+
+| lower |
++-------+
+| 37089 |
++-------+
+</codeblock>
+  </conbody>
+</concept>


[11/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_proxy.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_proxy.xml b/docs/topics/impala_proxy.xml
new file mode 100644
index 0000000..fc2e27c
--- /dev/null
+++ b/docs/topics/impala_proxy.xml
@@ -0,0 +1,635 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="proxy">
+
+  <title>Using Impala through a Proxy for High Availability</title>
+  <titlealts audience="PDF"><navtitle>Load-Balancing Proxy for HA</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="High Availability"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Network"/>
+      <data name="Category" value="Proxy"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      For most clusters that have multiple users and production availability requirements, you might set up a proxy
+      server to relay requests to and from Impala.
+    </p>
+
+    <p>
+      Currently, the Impala statestore mechanism does not include such proxying and load-balancing features. Set up
+      a software package of your choice to perform these functions.
+    </p>
+
+    <note>
+      <p conref="../shared/impala_common.xml#common/statestored_catalogd_ha_blurb"/>
+    </note>
+
+    <p outputclass="toc inpage"/>
+
+  </conbody>
+
+  <concept id="proxy_overview">
+
+    <title>Overview of Proxy Usage and Load Balancing for Impala</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Concepts"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        Using a load-balancing proxy server for Impala has the following advantages:
+      </p>
+
+      <ul>
+        <li>
+          Applications connect to a single well-known host and port, rather than keeping track of the hosts where
+          the <cmdname>impalad</cmdname> daemon is running.
+        </li>
+
+        <li>
+          If any host running the <cmdname>impalad</cmdname> daemon becomes unavailable, application connection
+          requests still succeed because you always connect to the proxy server rather than a specific host running
+          the <cmdname>impalad</cmdname> daemon.
+        </li>
+
+        <li>
+          The coordinator node for each Impala query potentially requires more memory and CPU cycles than the other
+          nodes that process the query. The proxy server can issue queries using round-robin scheduling, so that
+          each connection uses a different coordinator node. This load-balancing technique lets the Impala nodes
+          share this additional work, rather than concentrating it on a single machine.
+        </li>
+      </ul>
+
+      <p>
+        The following setup steps are a general outline that apply to any load-balancing proxy software:
+      </p>
+
+      <ol>
+        <li>
+          Download the load-balancing proxy software. It should only need to be installed and configured on a
+          single host. Pick a host other than the DataNodes where <cmdname>impalad</cmdname> is running,
+          because the intention is to protect against the possibility of one or more of these DataNodes becoming unavailable.
+        </li>
+
+        <li>
+          Configure the load balancer (typically by editing a configuration file).
+          In particular:
+          <ul>
+            <li>
+              <p>
+                Set up a port that the load balancer will listen on to relay Impala requests back and forth.
+              </p>
+            </li>
+            <li>
+              <p rev="DOCS-690">
+                Consider enabling <q>sticky sessions</q>. <ph rev="upstream">Cloudera</ph> recommends enabling this setting
+                so that stateless client applications such as <cmdname>impalad</cmdname> and Hue
+                are not disconnected from long-running queries. Evaluate whether this setting is
+                appropriate for your combination of workload and client applications.
+              </p>
+            </li>
+            <li>
+              <p>
+                For Kerberized clusters, follow the instructions in <xref href="impala_proxy.xml#proxy_kerberos"/>.
+              </p>
+            </li>
+          </ul>
+        </li>
+
+        <li>
+          Specify the host and port settings for each Impala node. These are the hosts that the load balancer will
+          choose from when relaying each Impala query. See <xref href="impala_ports.xml#ports"/> for when to use
+          port 21000, 21050, or another value depending on what type of connections you are load balancing.
+          <note rev="CDH-30399">
+            <p rev="CDH-30399">
+              In particular, if you are using Hue or JDBC-based applications,
+              you typically set up load balancing for both ports 21000 and 21050, because
+              these client applications connect through port 21050 while the <cmdname>impala-shell</cmdname>
+              command connects through port 21000.
+            </p>
+          </note>
+        </li>
+
+        <li>
+          Run the load-balancing proxy server, pointing it at the configuration file that you set up.
+        </li>
+
+        <li>
+          On systems managed by Cloudera Manager, on the page
+          <menucascade><uicontrol>Impala</uicontrol><uicontrol>Configuration</uicontrol><uicontrol>Impala Daemon
+          Default Group</uicontrol></menucascade>, specify a value for the <uicontrol>Impala Daemons Load
+          Balancer</uicontrol> field. Specify the address of the load balancer in
+          <codeph><varname>host</varname>:<varname>port</varname></codeph> format. This setting lets Cloudera
+          Manager route all appropriate Impala-related operations through the proxy server.
+        </li>
+
+        <li>
+          For any scripts, jobs, or configuration settings for applications that formerly connected to a specific
+          datanode to run Impala SQL statements, change the connection information (such as the <codeph>-i</codeph>
+          option in <cmdname>impala-shell</cmdname>) to point to the load balancer instead.
+        </li>
+      </ol>
+
+      <note>
+        The following sections use the HAProxy software as a representative example of a load balancer
+        that you can use with Impala.
+        For information specifically about using Impala with the F5 BIG-IP load balancer, see
+        <xref href="http://www.cloudera.com/documentation/other/reference-architecture/PDF/Impala-HA-with-F5-BIG-IP.pdf" scope="external" format="html">Impala HA with F5 BIG-IP</xref>.
+      </note>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="proxy_balancing" rev="CDH-33836 DOCS-349 CDH-39925 CDH-36812" audience="Cloudera">
+    <title>Choosing the Load-Balancing Algorithm</title>
+    <conbody>
+      <p>
+        Load-balancing software offers a number of algorithms to distribute requests.
+        Each algorithm has its own characteristics that make it suitable in some situations
+        but not others.
+      </p>
+
+      <dl>
+        <dlentry>
+          <dt>leastconn</dt>
+          <dd>
+            Connects sessions to the coordinator with the fewest connections, to balance the load evenly.
+            Typically used for workloads consisting of many independent, short-running queries.
+            In configurations with only a few client machines, this setting can avoid having all
+            requests go to only a small set of coordinators.
+          </dd>
+        </dlentry>
+        <dlentry>
+          <dt>source affinity</dt>
+          <dd>
+            Sessions from the same IP address always go to the same coordinator.
+            A good choice for Impala workloads containing a mix of queries and
+            DDL statements, such as <codeph>CREATE TABLE</codeph> and <codeph>ALTER TABLE</codeph>.
+            Because the metadata changes from a DDL statement take time to propagate across the cluster,
+            prefer to use source affinity in this case. If necessary, run the DDL and subsequent
+            queries that depend on the results of the DDL through the same session, for example
+            by running <codeph>impala-shell -f <varname>script_file</varname></codeph> to submit
+            several statements through a single session.
+            An alternative is to set the query option <codeph>SYNC_DDL=1</codeph>
+            to hold back subsequent queries until the results of a DDL operation have propagated
+            throughout the cluster, but that is a relatively expensive setting.
+            Recommended for use with Hue.
+          </dd>
+        </dlentry>
+        <dlentry>
+          <dt>sticky</dt>
+          <dd>
+            Similar to source affinity. Sessions from the same IP address always go to the same coordinator.
+            The maintenance overhead for the <q>stick tables</q> can cause long-running Hue sessions
+            to disconnect, therefore source affinity is often a better choice.
+          </dd>
+        </dlentry>
+        <dlentry>
+          <dt>round-robin</dt>
+          <dd>
+            Distributes connections to all coordinator nodes.
+            Typically not recommended for Impala.
+          </dd>
+        </dlentry>
+      </dl>
+
+      <p>
+        You might need to perform benchmarks and load testing to determine which setting is optimal for your
+        use case. If some client applications have special characteristics, such as long-running Hue queries
+        working best with source affinity, you might configure multiple virtual IP addresses with a
+        different load-balancing algorithm for each.
+      </p>
+
+    </conbody>
+  </concept>
+
+  <concept id="proxy_kerberos">
+
+    <title>Special Proxy Considerations for Clusters Using Kerberos</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="Kerberos"/>
+      <data name="Category" value="Authentication"/>
+      <data name="Category" value="Proxy"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        In a cluster using Kerberos, applications check host credentials to verify that the host they are
+        connecting to is the same one that is actually processing the request, to prevent man-in-the-middle
+        attacks. To clarify that the load-balancing proxy server is legitimate, perform these extra Kerberos setup
+        steps:
+      </p>
+
+      <ol>
+        <li>
+          This section assumes you are starting with a Kerberos-enabled cluster. See
+          <xref href="impala_kerberos.xml#kerberos"/> for instructions for setting up Impala with Kerberos. See the
+          <cite>CDH Security Guide</cite> for
+          <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_kerberos_prin_keytab_deploy.html" scope="external" format="html">general steps to set up Kerberos</xref>.
+        </li>
+
+        <li>
+          Choose the host you will use for the proxy server. Based on the Kerberos setup procedure, it should
+          already have an entry <codeph>impala/<varname>proxy_host</varname>@<varname>realm</varname></codeph> in
+          its keytab. If not, go back over the initial Kerberos configuration steps for the keytab on each host
+          running the <cmdname>impalad</cmdname> daemon.
+        </li>
+
+        <li rev="CDH-40363">
+          For a cluster managed by Cloudera Manager (5.4.2 or higher), fill in the Impala configuration setting
+          <uicontrol>Impala Daemons Load Balancer</uicontrol> with the appropriate host:port combination.
+          Then restart the Impala service.
+          For systems using a recent level of Cloudera Manager, this is all the configuration you need; you can skip the remaining steps in this procedure.
+        </li>
+
+        <li>
+          On systems not managed by Cloudera Manager, or systems using Cloudera Manager earlier than 5.4.2:
+
+        <ol>
+          <li>
+            Copy the keytab file from the proxy host to all other hosts in the cluster that run the
+            <cmdname>impalad</cmdname> daemon. (For optimal performance, <cmdname>impalad</cmdname> should be running
+            on all DataNodes in the cluster.) Put the keytab file in a secure location on each of these other hosts.
+          </li>
+
+          <li>
+            Add an entry <codeph>impala/<varname>actual_hostname</varname>@<varname>realm</varname></codeph> to the keytab on each
+            host running the <cmdname>impalad</cmdname> daemon.
+          </li>
+
+          <li>
+            For each impalad node, merge the existing keytab with the proxy\u2019s keytab using
+            <cmdname>ktutil</cmdname>, producing a new keytab file. For example:
+  <codeblock>$ ktutil
+  ktutil: read_kt proxy.keytab
+  ktutil: read_kt impala.keytab
+  ktutil: write_kt proxy_impala.keytab
+  ktutil: quit</codeblock>
+            <note>
+              On systems managed by Cloudera Manager 5.1.0 and later, the keytab merging happens automatically. To
+              verify that Cloudera Manager has merged the keytabs, run the command:
+  <codeblock>klist -k <varname>keytabfile</varname></codeblock>
+              which lists the credentials for both <codeph>principal</codeph> and <codeph>be_principal</codeph> on
+              all nodes.
+            </note>
+          </li>
+
+          <li>
+            Make sure that the <codeph>impala</codeph> user has permission to read this merged keytab file.
+          </li>
+
+          <li>
+            Change some configuration settings for each host in the cluster that participates in the load balancing.
+            Follow the appropriate steps depending on whether you use Cloudera Manager or not:
+            <ul>
+              <li> In the <cmdname>impalad</cmdname> option definition, or the advanced
+                configuration snippet, add: <codeblock>--principal=impala/<varname>proxy_host</varname>@<varname>realm</varname>
+  --be_principal=impala/<varname>actual_host</varname>@<varname>realm</varname>
+  --keytab_file=<varname>path_to_merged_keytab</varname></codeblock>
+                <note>
+                  <p>On a cluster managed by Cloudera Manager 5.1 (or higher),
+                    when you set up Kerberos authentication using the wizard, you
+                    can choose to allow Cloudera Manager to deploy the
+                      <systemoutput>krb5.conf</systemoutput> on your cluster. In
+                    such a case, you do not need to explicitly modify safety valve
+                    parameters as directed above. </p>
+                  <p>Every host has a different <codeph>--be_principal</codeph>
+                    because the actual hostname is different on each host. </p>
+                  <p> Specify the fully qualified domain name (FQDN) for the proxy
+                    host, not the IP address. Use the exact FQDN as returned by a
+                    reverse DNS lookup for the associated IP address. </p>
+                </note>
+              </li>
+
+              <li>
+                On a cluster managed by Cloudera Manager, create a role group to set the configuration values from
+                the preceding step on a per-host basis.
+              </li>
+
+              <li>
+                On a cluster not managed by Cloudera Manager, see
+                <xref href="impala_config_options.xml#config_options"/> for the procedure to modify the startup
+                options.
+              </li>
+            </ul>
+          </li>
+
+          <li>
+            Restart Impala to make the changes take effect. Follow the appropriate steps depending on whether you use
+            Cloudera Manager or not:
+            <ul>
+              <li>
+                On a cluster managed by Cloudera Manager, restart the Impala service.
+              </li>
+
+              <li>
+                On a cluster not managed by Cloudera Manager, restart the <cmdname>impalad</cmdname> daemons on all
+                hosts in the cluster, as well as the <cmdname>statestored</cmdname> and <cmdname>catalogd</cmdname>
+                daemons.
+              </li>
+            </ul>
+          </li>
+        </ol>
+        </li>
+      </ol>
+
+<!--
+We basically want to merge the keytab from the proxy host to all the impalad host's keytab file. To merge two keytab files, we first need to ship the proxy keytab to all the impalad node, then merge keytab files using MIT Kerberos "ktutil" command line tool.
+
+<codeblock>$ ktutil
+ktutil: read_kt krb5.keytab
+ktutil: read_kt proxy-host.keytab
+ktutil: write_kt krb5.keytab
+ktutil: quit</codeblock>
+
+The setup of the -principal and -be_principal has to be set through safety valve.
+-->
+
+    </conbody>
+
+  </concept>
+
+  <concept id="tut_proxy">
+
+    <title>Example of Configuring HAProxy Load Balancer for Impala</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Configuring"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        If you are not already using a load-balancing proxy, you can experiment with
+        <xref href="http://haproxy.1wt.eu/" scope="external" format="html">HAProxy</xref> a free, open source load
+        balancer. This example shows how you might install and configure that load balancer on a Red Hat Enterprise
+        Linux system.
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            Install the load balancer: <codeph>yum install haproxy</codeph>
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Set up the configuration file: <filepath>/etc/haproxy/haproxy.cfg</filepath>. See the following section
+            for a sample configuration file.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Run the load balancer (on a single host, preferably one not running <cmdname>impalad</cmdname>):
+          </p>
+<codeblock>/usr/sbin/haproxy \u2013f /etc/haproxy/haproxy.cfg</codeblock>
+        </li>
+
+        <li>
+          <p>
+            In <cmdname>impala-shell</cmdname>, JDBC applications, or ODBC applications, connect to the listener
+            port of the proxy host, rather than port 21000 or 21050 on a host actually running <cmdname>impalad</cmdname>.
+            The sample configuration file sets haproxy to listen on port 25003, therefore you would send all
+            requests to <codeph><varname>haproxy_host</varname>:25003</codeph>.
+          </p>
+        </li>
+      </ul>
+
+      <p>
+        This is the sample <filepath>haproxy.cfg</filepath> used in this example:
+      </p>
+
+<codeblock>global
+    # To have these messages end up in /var/log/haproxy.log you will
+    # need to:
+    #
+    # 1) configure syslog to accept network log events.  This is done
+    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
+    #    /etc/sysconfig/syslog
+    #
+    # 2) configure local2 events to go to the /var/log/haproxy.log
+    #   file. A line like the following can be added to
+    #   /etc/sysconfig/syslog
+    #
+    #    local2.*                       /var/log/haproxy.log
+    #
+    log         127.0.0.1 local0
+    log         127.0.0.1 local1 notice
+    chroot      /var/lib/haproxy
+    pidfile     /var/run/haproxy.pid
+    maxconn     4000
+    user        haproxy
+    group       haproxy
+    daemon
+
+    # turn on stats unix socket
+    #stats socket /var/lib/haproxy/stats
+
+#---------------------------------------------------------------------
+# common defaults that all the 'listen' and 'backend' sections will
+# use if not designated in their block
+#
+# You might need to adjust timing values to prevent timeouts.
+#---------------------------------------------------------------------
+defaults
+    mode                    http
+    log                     global
+    option                  httplog
+    option                  dontlognull
+    option http-server-close
+    option forwardfor       except 127.0.0.0/8
+    option                  redispatch
+    retries                 3
+    maxconn                 3000
+    contimeout 5000
+    clitimeout 50000
+    srvtimeout 50000
+
+#
+# This sets up the admin page for HA Proxy at port 25002.
+#
+listen stats :25002
+    balance
+    mode http
+    stats enable
+    stats auth <varname>username</varname>:<varname>password</varname>
+
+# This is the setup for Impala. Impala client connect to load_balancer_host:25003.
+# HAProxy will balance connections among the list of servers listed below.
+# The list of Impalad is listening at port 21000 for beeswax (impala-shell) or original ODBC driver.
+# For JDBC or ODBC version 2.x driver, use port 21050 instead of 21000.
+listen impala :25003
+    mode tcp
+    option tcplog
+    balance leastconn
+
+    server <varname>symbolic_name_1</varname> impala-host-1.example.com:21000
+    server <varname>symbolic_name_2</varname> impala-host-2.example.com:21000
+    server <varname>symbolic_name_3</varname> impala-host-3.example.com:21000
+    server <varname>symbolic_name_4</varname> impala-host-4.example.com:21000
+
+# Setup for Hue or other JDBC-enabled applications.
+# In particular, Hue requires sticky sessions.
+# The application connects to load_balancer_host:21051, and HAProxy balances
+# connections to the associated hosts, where Impala listens for JDBC
+# requests on port 21050.
+listen impalajdbc :21051
+    mode tcp
+    option tcplog
+    balance source
+    server <varname>symbolic_name_5</varname> impala-host-1.example.com:21050
+    server <varname>symbolic_name_6</varname> impala-host-2.example.com:21050
+    server <varname>symbolic_name_7</varname> impala-host-3.example.com:21050
+    server <varname>symbolic_name_8</varname> impala-host-4.example.com:21050
+</codeblock>
+
+      <note conref="../shared/impala_common.xml#common/proxy_jdbc_caveat"/>
+
+      <p audience="Cloudera">
+        The following example shows extra steps needed for a cluster using Kerberos authentication:
+      </p>
+
+<codeblock audience="Cloudera">$ klist
+$ impala-shell -k
+$ kinit -r 1d -kt /systest/keytabs/hdfs.keytab hdfs
+$ impala-shell -i c2104.hal.cloudera.com:21000
+$ impala-shell -i c2104.hal.cloudera.com:25003
+[root@c2104 alan]# ps -ef |grep impalad
+root      6442  6428  0 12:21 pts/0    00:00:00 grep impalad
+impala   30577 22192 99 Nov14 ?        3-16:42:32 /usr/lib/impala/sbin-debug/impalad --flagfile=/var/run/cloudera-scm-agent/process/10342-impala-IMPALAD/impala-conf/impalad_flags
+[root@c2104 alan]# vi /var/run/cloudera-scm-agent/process/10342-impala-IMPALAD/impala-conf/impalad_flags
+$ klist -k /var/run/cloudera-scm-agent/process/10342-impala-IMPALAD/impala.keytab
+Keytab name: FILE:/var/run/cloudera-scm-agent/process/10342-impala-IMPALAD/impala.keytab
+KVNO Principal
+---- --------------------------------------------------------------------------
+   2 impala/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 impala/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 impala/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 impala/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 HTTP/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 HTTP/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 HTTP/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 HTTP/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+$ klist
+Ticket cache: FILE:/tmp/krb5cc_4028
+Default principal: hdfs@HAL17.CLOUDERA.COM
+
+Valid starting     Expires            Service principal
+11/15/13 12:17:17  11/15/13 12:32:17  krbtgt/HAL17.CLOUDERA.COM@HAL17.CLOUDERA.COM
+        renew until 11/16/13 12:17:17
+11/15/13 12:17:21  11/15/13 12:32:17  impala/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+        renew until 11/16/13 12:17:17
+$ kinit -r 1d -kt /systest/keytabs/hdfs.keytab hdfs
+$ kinit -R
+$ impala-shell -k -i c2106.hal.cloudera.com:21000
+Starting Impala Shell using Kerberos authentication
+Using service name 'impala'
+Connected to c2106.hal.cloudera.com:21000
+$ impala-shell -i c2104.hal.cloudera.com:25003
+$ impala-shell -k -i c2104.hal.cloudera.com:25003
+Starting Impala Shell using Kerberos authentication
+Using service name 'impala'
+Connected to c2104.hal.cloudera.com:25003
+[c2104.hal.cloudera.com:25003] &gt; create table alan_tmp(a int);
+Query: create table alan_tmp(a int)
+ERROR: InternalException: Got exception: org.apache.hadoop.ipc.RemoteException User: hive/c2102.hal.cloudera.com@HAL17.CLOUDERA.COM is not allowed to impersonate impala/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+$ kdestroy
+$ kinit -r 1d -kt /systest/keytabs/hdfs.keytab hdfs
+$ impala-shell -k -i c2104.hal.cloudera.com:25003
+# klist -k c2104.keytab
+Keytab name: FILE:c2104.keytab
+KVNO Principal
+---- --------------------------------------------------------------------------
+   2 impala/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 impala/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 impala/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 impala/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 HTTP/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 HTTP/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 HTTP/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 HTTP/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+$ klist -k -t c2106.keytab
+Keytab name: FILE:c2106.keytab
+KVNO Timestamp         Principal
+---- ----------------- --------------------------------------------------------
+   2 02/14/13 12:12:22 HTTP/c2106.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 02/14/13 12:12:22 HTTP/c2106.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 02/14/13 12:12:22 HTTP/c2106.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 02/14/13 12:12:22 HTTP/c2106.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 02/14/13 12:12:22 impala/c2106.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 02/14/13 12:12:22 impala/c2106.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 02/14/13 12:12:22 impala/c2106.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 02/14/13 12:12:22 impala/c2106.hal.cloudera.com@HAL17.CLOUDERA.COM
+$ ktutil
+ktutil:  rkt c2104.keytab
+ktutil:  rkt c2106.keytab
+ktutil:  wkt my_test.keytab
+ktutil:  q
+$ klist -k -t my_test.keytab
+Keytab name: FILE:my_test.keytab
+KVNO Timestamp         Principal
+---- ----------------- --------------------------------------------------------
+   2 11/21/13 16:22:40 impala/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 11/21/13 16:22:40 impala/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 11/21/13 16:22:40 impala/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 11/21/13 16:22:40 impala/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 11/21/13 16:22:40 HTTP/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 11/21/13 16:22:40 HTTP/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 11/21/13 16:22:40 HTTP/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 11/21/13 16:22:40 HTTP/c2104.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 11/21/13 16:22:40 HTTP/c2106.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 11/21/13 16:22:41 HTTP/c2106.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 11/21/13 16:22:41 HTTP/c2106.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 11/21/13 16:22:41 HTTP/c2106.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 11/21/13 16:22:41 impala/c2106.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 11/21/13 16:22:41 impala/c2106.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 11/21/13 16:22:41 impala/c2106.hal.cloudera.com@HAL17.CLOUDERA.COM
+   2 11/21/13 16:22:41 impala/c2106.hal.cloudera.com@HAL17.CLOUDERA.COM
+$ kdestroy
+$ kinit -r 1d -kt /systest/keytabs/hdfs.keytab hdfs
+$ vi README
+$ kinit -R
+$ impala-shell -k -i c2104.hal.cloudera.com:25003
+Starting Impala Shell using Kerberos authentication
+Using service name 'impala'
+Connected to c2104.hal.cloudera.com:25003
+<ph conref="../shared/ImpalaVariables.xml#impala_vars/ImpaladBanner"/>
+Welcome to the Impala shell. Press TAB twice to see a list of available commands.
+
+Copyright (c) 2012 Cloudera, Inc. All rights reserved.
+
+<ph conref="../shared/ImpalaVariables.xml#impala_vars/ShellBanner"/>
+[c2104.hal.cloudera.com:25003] &gt; show tables;
+Query: show tables
+ERROR: AnalysisException: This Impala daemon is not ready to accept user requests. Status: Waiting for catalog update from the StateStore.
+[c2104.hal.cloudera.com:25003] &gt; quit;</codeblock>
+
+      <!--
+        At that point in the walkthrough with Alan Choi, we could never get Impala to accept any requests through the catalog server.
+        So I have not seen a 100% successful proxy setup process to verify all the details.
+      -->
+
+    </conbody>
+
+  </concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_query_lifetime.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_query_lifetime.xml b/docs/topics/impala_query_lifetime.xml
new file mode 100644
index 0000000..2f46d21
--- /dev/null
+++ b/docs/topics/impala_query_lifetime.xml
@@ -0,0 +1,31 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="query_lifetime">
+
+  <title>Impala Query Lifetime</title>
+
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Concepts"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Impala queries progress through a series of stages from the time they are initiated to the time
+      they are completed. A query can also be cancelled before it is entirely finished, either
+      because of an explicit cancellation, or because of a timeout, out-of-memory, or other error condition.
+      Understanding the query lifecycle can help you manage the throughput and resource usage of Impala
+      queries, especially in a high-concurrency or multi-workload environment.
+    </p>
+
+    <p outputclass="toc"/>
+  </conbody>
+
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_query_options.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_query_options.xml b/docs/topics/impala_query_options.xml
new file mode 100644
index 0000000..7ed6ac2
--- /dev/null
+++ b/docs/topics/impala_query_options.xml
@@ -0,0 +1,77 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="query_options">
+
+  <title>Query Options for the SET Statement</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="impala-shell"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Configuring"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      You can specify the following options using the <codeph>SET</codeph> statement, and those settings affect all
+      queries issued from that session.
+    </p>
+
+    <p>
+      Some query options are useful in day-to-day operations for improving usability, performance, or flexibility.
+    </p>
+
+    <p>
+      Other query options control special-purpose aspects of Impala operation and are intended primarily for
+      advanced debugging or troubleshooting.
+    </p>
+
+    <p>
+      Options with Boolean parameters can be set to 1 or <codeph>true</codeph> to enable, or 0 or <codeph>false</codeph>
+      to turn off.
+    </p>
+
+    <note rev="2.0.0">
+      <p rev="2.0.0">
+        In Impala 2.0 and later, you can set query options directly through the JDBC and ODBC interfaces by using the
+        <codeph>SET</codeph> statement. Formerly, <codeph>SET</codeph> was only available as a command within the
+        <cmdname>impala-shell</cmdname> interpreter.
+      </p>
+    </note>
+
+<!-- This is the list including defaults from the pre-release 1.2 impala-shell:
+        ABORT_ON_DEFAULT_LIMIT_EXCEEDED: 0
+        ABORT_ON_ERROR: 0
+        ALLOW_UNSUPPORTED_FORMATS: 0
+        BATCH_SIZE: 0
+        DEBUG_ACTION:
+        DEFAULT_ORDER_BY_LIMIT: -1
+        DISABLE_CODEGEN: 0
+        HBASE_CACHE_BLOCKS: 0
+        HBASE_CACHING: 0
+        MAX_ERRORS: 0
+        MAX_IO_BUFFERS: 0
+        MAX_SCAN_RANGE_LENGTH: 0
+        MEM_LIMIT: 0
+        NUM_NODES: 0
+        NUM_SCANNER_THREADS: 0
+        PARQUET_COMPRESSION_CODEC: SNAPPY
+        PARQUET_FILE_SIZE: 0
+        SUPPORT_START_OVER: false
+-->
+
+    <p outputclass="toc"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_set.xml#set"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_query_timeout_s.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_query_timeout_s.xml b/docs/topics/impala_query_timeout_s.xml
new file mode 100644
index 0000000..0486e01
--- /dev/null
+++ b/docs/topics/impala_query_timeout_s.xml
@@ -0,0 +1,56 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="2.0.0" id="query_timeout_s">
+
+  <title>QUERY_TIMEOUT_S Query Option (<keyword keyref="impala20"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>QUERY_TIMEOUT_S</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.0.0">
+      <indexterm audience="Cloudera">QUERY_TIMEOUT_S query option</indexterm>
+      Sets the idle query timeout value for the session, in seconds. Queries that sit idle for longer than the
+      timeout value are automatically cancelled. If the system administrator specified the
+      <codeph>--idle_query_timeout</codeph> startup option, <codeph>QUERY_TIMEOUT_S</codeph> must be smaller than
+      or equal to the <codeph>--idle_query_timeout</codeph> value.
+    </p>
+
+    <note conref="../shared/impala_common.xml#common/timeout_clock_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>SET QUERY_TIMEOUT_S=<varname>seconds</varname>;</codeblock>
+
+<!-- Don't have a compelling example to show at this time because the 'idle' aspect only applies
+     when the client is careless and leaves the query open. Can't easily demonstrate in impala-shell.
+
+     <p conref="../shared/impala_common.xml#common/example_blurb"/>
+-->
+
+    <p>
+      <b>Type:</b> numeric
+    </p>
+
+    <p>
+      <b>Default:</b> 0 (no timeout if <codeph>--idle_query_timeout</codeph> not in effect; otherwise, use
+      <codeph>--idle_query_timeout</codeph> value)
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/added_in_20"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_timeouts.xml#timeouts"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_rcfile.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_rcfile.xml b/docs/topics/impala_rcfile.xml
new file mode 100644
index 0000000..1bfab8c
--- /dev/null
+++ b/docs/topics/impala_rcfile.xml
@@ -0,0 +1,244 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="rcfile">
+
+  <title>Using the RCFile File Format with Impala Tables</title>
+  <titlealts audience="PDF"><navtitle>RCFile Data Files</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <!-- <data name="Category" value="RCFile"/> -->
+      <data name="Category" value="File Formats"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">RCFile support in Impala</indexterm>
+      Impala supports using RCFile data files.
+    </p>
+
+    <table>
+      <title>RCFile Format Support in Impala</title>
+      <tgroup cols="5">
+        <colspec colname="1" colwidth="10*"/>
+        <colspec colname="2" colwidth="10*"/>
+        <colspec colname="3" colwidth="20*"/>
+        <colspec colname="4" colwidth="30*"/>
+        <colspec colname="5" colwidth="30*"/>
+        <thead>
+          <row>
+            <entry>
+              File Type
+            </entry>
+            <entry>
+              Format
+            </entry>
+            <entry>
+              Compression Codecs
+            </entry>
+            <entry>
+              Impala Can CREATE?
+            </entry>
+            <entry>
+              Impala Can INSERT?
+            </entry>
+          </row>
+        </thead>
+        <tbody>
+          <row conref="impala_file_formats.xml#file_formats/rcfile_support">
+            <entry/>
+          </row>
+        </tbody>
+      </tgroup>
+    </table>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="rcfile_create">
+
+    <title>Creating RCFile Tables and Loading Data</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="ETL"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        If you do not have an existing data file to use, begin by creating one in the appropriate format.
+      </p>
+
+      <p>
+        <b>To create an RCFile table:</b>
+      </p>
+
+      <p>
+        In the <codeph>impala-shell</codeph> interpreter, issue a command similar to:
+      </p>
+
+<codeblock>create table rcfile_table (<varname>column_specs</varname>) stored as rcfile;</codeblock>
+
+      <p>
+        Because Impala can query some kinds of tables that it cannot currently write to, after creating tables of
+        certain file formats, you might use the Hive shell to load the data. See
+        <xref href="impala_file_formats.xml#file_formats"/> for details. After loading data into a table through
+        Hive or other mechanism outside of Impala, issue a <codeph>REFRESH <varname>table_name</varname></codeph>
+        statement the next time you connect to the Impala node, before querying the table, to make Impala recognize
+        the new data.
+      </p>
+
+      <note type="important">
+        See <xref href="impala_known_issues.xml#known_issues"/> for potential compatibility issues with
+        RCFile tables created in Hive 0.12, due to a change in the default RCFile SerDe for Hive.
+      </note>
+
+      <p>
+        For example, here is how you might create some RCFile tables in Impala (by specifying the columns
+        explicitly, or cloning the structure of another table), load data through Hive, and query them through
+        Impala:
+      </p>
+
+<codeblock>$ impala-shell -i localhost
+[localhost:21000] &gt; create table rcfile_table (x int) stored as rcfile;
+[localhost:21000] &gt; create table rcfile_clone like some_other_table stored as rcfile;
+[localhost:21000] &gt; quit;
+
+$ hive
+hive&gt; insert into table rcfile_table select x from some_other_table;
+3 Rows loaded to rcfile_table
+Time taken: 19.015 seconds
+hive&gt; quit;
+
+$ impala-shell -i localhost
+[localhost:21000] &gt; select * from rcfile_table;
+Returned 0 row(s) in 0.23s
+[localhost:21000] &gt; -- Make Impala recognize the data loaded through Hive;
+[localhost:21000] &gt; refresh rcfile_table;
+[localhost:21000] &gt; select * from rcfile_table;
++---+
+| x |
++---+
+| 1 |
+| 2 |
+| 3 |
++---+
+Returned 3 row(s) in 0.23s</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_unsupported_filetype"/>
+
+    </conbody>
+  </concept>
+
+  <concept id="rcfile_compression">
+
+    <title>Enabling Compression for RCFile Tables</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Snappy"/>
+      <data name="Category" value="Compression"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">compression</indexterm>
+        You may want to enable compression on existing tables. Enabling compression provides performance gains in
+        most cases and is supported for RCFile tables. For example, to enable Snappy compression, you would specify
+        the following additional settings when loading data through the Hive shell:
+      </p>
+
+<codeblock>hive&gt; SET hive.exec.compress.output=true;
+hive&gt; SET mapred.max.split.size=256000000;
+hive&gt; SET mapred.output.compression.type=BLOCK;
+hive&gt; SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
+hive&gt; INSERT OVERWRITE TABLE <varname>new_table</varname> SELECT * FROM <varname>old_table</varname>;</codeblock>
+
+      <p>
+        If you are converting partitioned tables, you must complete additional steps. In such a case, specify
+        additional settings similar to the following:
+      </p>
+
+<codeblock>hive&gt; CREATE TABLE <varname>new_table</varname> (<varname>your_cols</varname>) PARTITIONED BY (<varname>partition_cols</varname>) STORED AS <varname>new_format</varname>;
+hive&gt; SET hive.exec.dynamic.partition.mode=nonstrict;
+hive&gt; SET hive.exec.dynamic.partition=true;
+hive&gt; INSERT OVERWRITE TABLE <varname>new_table</varname> PARTITION(<varname>comma_separated_partition_cols</varname>) SELECT * FROM <varname>old_table</varname>;</codeblock>
+
+      <p>
+        Remember that Hive does not require that you specify a source format for it. Consider the case of
+        converting a table with two partition columns called <codeph>year</codeph> and <codeph>month</codeph> to a
+        Snappy compressed RCFile. Combining the components outlined previously to complete this table conversion,
+        you would specify settings similar to the following:
+      </p>
+
+<codeblock>hive&gt; CREATE TABLE tbl_rc (int_col INT, string_col STRING) STORED AS RCFILE;
+hive&gt; SET hive.exec.compress.output=true;
+hive&gt; SET mapred.max.split.size=256000000;
+hive&gt; SET mapred.output.compression.type=BLOCK;
+hive&gt; SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
+hive&gt; SET hive.exec.dynamic.partition.mode=nonstrict;
+hive&gt; SET hive.exec.dynamic.partition=true;
+hive&gt; INSERT OVERWRITE TABLE tbl_rc SELECT * FROM tbl;</codeblock>
+
+      <p>
+        To complete a similar process for a table that includes partitions, you would specify settings similar to
+        the following:
+      </p>
+
+<codeblock>hive&gt; CREATE TABLE tbl_rc (int_col INT, string_col STRING) PARTITIONED BY (year INT) STORED AS RCFILE;
+hive&gt; SET hive.exec.compress.output=true;
+hive&gt; SET mapred.max.split.size=256000000;
+hive&gt; SET mapred.output.compression.type=BLOCK;
+hive&gt; SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
+hive&gt; SET hive.exec.dynamic.partition.mode=nonstrict;
+hive&gt; SET hive.exec.dynamic.partition=true;
+hive&gt; INSERT OVERWRITE TABLE tbl_rc PARTITION(year) SELECT * FROM tbl;</codeblock>
+
+      <note>
+        <p>
+          The compression type is specified in the following command:
+        </p>
+<codeblock>SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;</codeblock>
+        <p>
+          You could elect to specify alternative codecs such as <codeph>GzipCodec</codeph> here.
+        </p>
+      </note>
+    </conbody>
+  </concept>
+
+  <concept id="rcfile_performance">
+
+    <title>Query Performance for Impala RCFile Tables</title>
+
+    <conbody>
+
+      <p>
+        In general, expect query performance with RCFile tables to be
+        faster than with tables using text data, but slower than with
+        Parquet tables. See <xref href="impala_parquet.xml#parquet"/>
+        for information about using the Parquet file format for
+        high-performance analytic queries.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/s3_block_splitting"/>
+
+    </conbody>
+  </concept>
+
+  <concept audience="Cloudera" id="rcfile_data_types">
+
+    <title>Data Type Considerations for RCFile Tables</title>
+
+    <conbody>
+
+      <p></p>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_real.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_real.xml b/docs/topics/impala_real.xml
new file mode 100644
index 0000000..12ef5aa
--- /dev/null
+++ b/docs/topics/impala_real.xml
@@ -0,0 +1,46 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="real">
+
+  <title>REAL Data Type</title>
+  <titlealts audience="PDF"><navtitle>REAL</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      An alias for the <codeph>DOUBLE</codeph> data type. See <xref href="impala_double.xml#double"/> for details.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      These examples show how you can use the type names <codeph>REAL</codeph> and <codeph>DOUBLE</codeph>
+      interchangeably, and behind the scenes Impala treats them always as <codeph>DOUBLE</codeph>.
+    </p>
+
+<codeblock>[localhost:21000] &gt; create table r1 (x real);
+[localhost:21000] &gt; describe r1;
++------+--------+---------+
+| name | type   | comment |
++------+--------+---------+
+| x    | double |         |
++------+--------+---------+
+[localhost:21000] &gt; insert into r1 values (1.5), (cast (2.2 as double));
+[localhost:21000] &gt; select cast (1e6 as real);
++---------------------------+
+| cast(1000000.0 as double) |
++---------------------------+
+| 1000000                   |
++---------------------------+</codeblock>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_refresh.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_refresh.xml b/docs/topics/impala_refresh.xml
new file mode 100644
index 0000000..4b038ff
--- /dev/null
+++ b/docs/topics/impala_refresh.xml
@@ -0,0 +1,324 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="refresh">
+
+  <title>REFRESH Statement</title>
+  <titlealts audience="PDF"><navtitle>REFRESH</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Hive"/>
+      <data name="Category" value="Metastore"/>
+      <data name="Category" value="ETL"/>
+      <data name="Category" value="Ingest"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">REFRESH statement</indexterm>
+      To accurately respond to queries, the Impala node that acts as the coordinator (the node to which you are
+      connected through <cmdname>impala-shell</cmdname>, JDBC, or ODBC) must have current metadata about those
+      databases and tables that are referenced in Impala queries. If you are not familiar with the way Impala uses
+      metadata and how it shares the same metastore database as Hive, see
+      <xref href="impala_hadoop.xml#intro_metastore"/> for background information.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock rev="IMPALA-1683 CDH-43732">REFRESH [<varname>db_name</varname>.]<varname>table_name</varname> [PARTITION (<varname>key_col1</varname>=<varname>val1</varname> [, <varname>key_col2</varname>=<varname>val2</varname>...])]</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      Use the <codeph>REFRESH</codeph> statement to load the latest metastore metadata and block location data for
+      a particular table in these scenarios:
+    </p>
+
+    <ul>
+      <li>
+        After loading new data files into the HDFS data directory for the table. (Once you have set up an ETL
+        pipeline to bring data into Impala on a regular basis, this is typically the most frequent reason why
+        metadata needs to be refreshed.)
+      </li>
+
+      <li>
+        After issuing <codeph>ALTER TABLE</codeph>, <codeph>INSERT</codeph>, <codeph>LOAD DATA</codeph>, or other
+        table-modifying SQL statement in Hive.
+      </li>
+    </ul>
+
+    <note rev="2.3.0">
+      <p rev="2.3.0">
+        In <keyword keyref="impala23_full"/> and higher, the syntax <codeph>ALTER TABLE <varname>table_name</varname> RECOVER PARTITIONS</codeph>
+        is a faster alternative to <codeph>REFRESH</codeph> when the only change to the table data is the addition of
+        new partition directories through Hive or manual HDFS operations.
+        See <xref href="impala_alter_table.xml#alter_table"/> for details.
+      </p>
+    </note>
+
+    <p>
+      You only need to issue the <codeph>REFRESH</codeph> statement on the node to which you connect to issue
+      queries. The coordinator node divides the work among all the Impala nodes in a cluster, and sends read
+      requests for the correct HDFS blocks without relying on the metadata on the other nodes.
+    </p>
+
+    <p>
+      <codeph>REFRESH</codeph> reloads the metadata for the table from the metastore database, and does an
+      incremental reload of the low-level block location data to account for any new data files added to the HDFS
+      data directory for the table. It is a low-overhead, single-table operation, specifically tuned for the common
+      scenario where new data files are added to HDFS.
+    </p>
+
+    <p>
+      Only the metadata for the specified table is flushed. The table must already exist and be known to Impala,
+      either because the <codeph>CREATE TABLE</codeph> statement was run in Impala rather than Hive, or because a
+      previous <codeph>INVALIDATE METADATA</codeph> statement caused Impala to reload its entire metadata catalog.
+    </p>
+
+    <note>
+      <p rev="1.2">
+        The catalog service broadcasts any changed metadata as a result of Impala
+        <codeph>ALTER TABLE</codeph>, <codeph>INSERT</codeph> and <codeph>LOAD DATA</codeph> statements to all
+        Impala nodes. Thus, the <codeph>REFRESH</codeph> statement is only required if you load data through Hive
+        or by manipulating data files in HDFS directly. See <xref href="impala_components.xml#intro_catalogd"/> for
+        more information on the catalog service.
+      </p>
+      <p rev="1.2.1">
+        Another way to avoid inconsistency across nodes is to enable the
+        <codeph>SYNC_DDL</codeph> query option before performing a DDL statement or an <codeph>INSERT</codeph> or
+        <codeph>LOAD DATA</codeph>.
+      </p>
+      <p rev="1.1">
+        The table name is a required parameter. To flush the metadata for all tables, use the
+        <codeph><xref href="impala_invalidate_metadata.xml#invalidate_metadata">INVALIDATE METADATA</xref></codeph>
+        command.
+      </p>
+      <p conref="../shared/impala_common.xml#common/invalidate_then_refresh"/>
+    </note>
+
+    <p conref="../shared/impala_common.xml#common/refresh_vs_invalidate"/>
+
+    <p>
+      A metadata update for an <codeph>impalad</codeph> instance <b>is</b> required if:
+    </p>
+
+    <ul>
+      <li>
+        A metadata change occurs.
+      </li>
+
+      <li>
+        <b>and</b> the change is made through Hive.
+      </li>
+
+      <li>
+        <b>and</b> the change is made to a metastore database to which clients such as the Impala shell or ODBC directly
+        connect.
+      </li>
+    </ul>
+
+    <p rev="1.2">
+      A metadata update for an Impala node is <b>not</b> required after you run <codeph>ALTER TABLE</codeph>,
+      <codeph>INSERT</codeph>, or other table-modifying statement in Impala rather than Hive. Impala handles the
+      metadata synchronization automatically through the catalog service.
+    </p>
+
+    <p>
+      Database and table metadata is typically modified by:
+    </p>
+
+    <ul>
+      <li>
+        Hive - through <codeph>ALTER</codeph>, <codeph>CREATE</codeph>, <codeph>DROP</codeph> or
+        <codeph>INSERT</codeph> operations.
+      </li>
+
+      <li>
+        Impalad - through <codeph>CREATE TABLE</codeph>, <codeph>ALTER TABLE</codeph>, and <codeph>INSERT</codeph>
+        operations. <ph rev="1.2">Such changes are propagated to all Impala nodes by the
+        Impala catalog service.</ph>
+      </li>
+    </ul>
+
+    <p>
+      <codeph>REFRESH</codeph> causes the metadata for that table to be immediately reloaded. For a huge table,
+      that process could take a noticeable amount of time; but doing the refresh up front avoids an unpredictable
+      delay later, for example if the next reference to the table is during a benchmark test.
+    </p>
+
+    <p rev="IMPALA-1683 CDH-43732">
+      <b>Refreshing a single partition:</b>
+    </p>
+
+    <p rev="IMPALA-1683 CDH-43732">
+      In <keyword keyref="impala27_full"/> and higher, the <codeph>REFRESH</codeph> statement can apply to a single partition at a time,
+      rather than the whole table. Include the optional <codeph>PARTITION (<varname>partition_spec</varname>)</codeph>
+      clause and specify values for each of the partition key columns.
+    </p>
+
+    <p rev="IMPALA-1683 CDH-43732">
+      The following examples show how to make Impala aware of data added to a single partition, after data is loaded into
+      a partition's data directory using some mechanism outside Impala, such as Hive or Spark. The partition can be one that
+      Impala created and is already aware of, or a new partition created through Hive.
+    </p>
+
+<codeblock rev="IMPALA-1683 CDH-43732"><![CDATA[
+impala> create table p (x int) partitioned by (y int);
+impala> insert into p (x,y) values (1,2), (2,2), (2,1);
+impala> show partitions p;
++-------+-------+--------+------+...
+| y     | #Rows | #Files | Size |...
++-------+-------+--------+------+...
+| 1     | -1    | 1      | 2B   |...
+| 2     | -1    | 1      | 4B   |...
+| Total | -1    | 2      | 6B   |...
++-------+-------+--------+------+...
+
+-- ... Data is inserted into one of the partitions by some external mechanism ...
+beeline> insert into p partition (y = 1) values(1000);
+
+impala> refresh p partition (y=1);
+impala> select x from p where y=1;
++------+
+| x    |
++------+
+| 2    | <- Original data created by Impala
+| 1000 | <- Additional data inserted through Beeline
++------+
+]]>
+</codeblock>
+
+    <p rev="IMPALA-1683 CDH-43732">
+      The same applies for tables with more than one partition key column.
+      The <codeph>PARTITION</codeph> clause of the <codeph>REFRESH</codeph>
+      statement must include all the partition key columns.
+    </p>
+
+<codeblock rev="IMPALA-1683 CDH-43732"><![CDATA[
+impala> create table p2 (x int) partitioned by (y int, z int);
+impala> insert into p2 (x,y,z) values (0,0,0), (1,2,3), (2,2,3);
+impala> show partitions p2;
++-------+---+-------+--------+------+...
+| y     | z | #Rows | #Files | Size |...
++-------+---+-------+--------+------+...
+| 0     | 0 | -1    | 1      | 2B   |...
+| 2     | 3 | -1    | 1      | 4B   |...
+| Total |   | -1    | 2      | 6B   |...
++-------+---+-------+--------+------+...
+
+-- ... Data is inserted into one of the partitions by some external mechanism ...
+beeline> insert into p2 partition (y = 2, z = 3) values(1000);
+
+impala> refresh p2 partition (y=2, z=3);
+impala> select x from p where y=2 and z = 3;
++------+
+| x    |
++------+
+| 1    | <- Original data created by Impala
+| 2    | <- Original data created by Impala
+| 1000 | <- Additional data inserted through Beeline
++------+
+]]>
+</codeblock>
+
+    <p rev="IMPALA-1683 CDH-43732">
+      The following examples show how specifying a nonexistent partition does not cause any error,
+      and the order of the partition key columns does not have to match the column order in the table.
+      The partition spec must include all the partition key columns; specifying an incomplete set of
+      columns does cause an error.
+    </p>
+
+<codeblock rev="IMPALA-1683 CDH-43732"><![CDATA[
+-- Partition doesn't exist.
+refresh p2 partition (y=0, z=3);
+refresh p2 partition (y=0, z=-1)
+-- Key columns specified in a different order than the table definition.
+refresh p2 partition (z=1, y=0)
+-- Incomplete partition spec causes an error.
+refresh p2 partition (y=0)
+ERROR: AnalysisException: Items in partition spec must exactly match the partition columns in the table definition: default.p2 (1 vs 2)
+]]>
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/sync_ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      The following example shows how you might use the <codeph>REFRESH</codeph> statement after manually adding
+      new HDFS data files to the Impala data directory for a table:
+    </p>
+
+<codeblock>[impalad-host:21000] &gt; refresh t1;
+[impalad-host:21000] &gt; refresh t2;
+[impalad-host:21000] &gt; select * from t1;
+...
+[impalad-host:21000] &gt; select * from t2;
+... </codeblock>
+
+    <p>
+      For more examples of using <codeph>REFRESH</codeph> and <codeph>INVALIDATE METADATA</codeph> with a
+      combination of Impala and Hive operations, see <xref href="impala_tutorial.xml#tutorial_impala_hive"/>.
+    </p>
+
+    <p>
+      <b>Related impala-shell options:</b>
+    </p>
+
+    <p rev="1.1">
+      The <cmdname>impala-shell</cmdname> option <codeph>-r</codeph> issues an <codeph>INVALIDATE METADATA</codeph> statement
+      when starting up the shell, effectively performing a <codeph>REFRESH</codeph> of all tables.
+      Due to the expense of reloading the metadata for all tables, the <cmdname>impala-shell</cmdname> <codeph>-r</codeph>
+      option is not recommended for day-to-day use in a production environment. (This option was mainly intended as a workaround
+      for synchronization issues in very old Impala versions.)
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+    <p rev="CDH-19187">
+      The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+      typically the <codeph>impala</codeph> user, must have execute
+      permissions for all the relevant directories holding table data.
+      (A table could have data spread across multiple directories,
+      or in unexpected paths, if it uses partitioning or
+      specifies a <codeph>LOCATION</codeph> attribute for
+      individual partitions or the entire table.)
+      Issues with permissions might not cause an immediate error for this statement,
+      but subsequent statements such as <codeph>SELECT</codeph>
+      or <codeph>SHOW TABLE STATS</codeph> could fail.
+    </p>
+    <p rev="IMPALA-1683 CDH-43732">
+      All HDFS and Sentry permissions and privileges are the same whether you refresh the entire table
+      or a single partition.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/hdfs_blurb"/>
+
+    <p>
+      The <codeph>REFRESH</codeph> command checks HDFS permissions of the underlying data files and directories,
+      caching this information so that a statement can be cancelled immediately if for example the
+      <codeph>impala</codeph> user does not have permission to write to the data directory for the table. Impala
+      reports any lack of write permissions as an <codeph>INFO</codeph> message in the log file, in case that
+      represents an oversight. If you change HDFS permissions to make data readable or writeable by the Impala
+      user, issue another <codeph>REFRESH</codeph> to make Impala aware of the change.
+    </p>
+
+    <note conref="../shared/impala_common.xml#common/compute_stats_next"/>
+
+    <p conref="../shared/impala_common.xml#common/s3_blurb"/>
+    <p conref="../shared/impala_common.xml#common/s3_metadata"/>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_hadoop.xml#intro_metastore"/>,
+      <xref href="impala_invalidate_metadata.xml#invalidate_metadata"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_release_notes.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_release_notes.xml b/docs/topics/impala_release_notes.xml
new file mode 100644
index 0000000..65a3997
--- /dev/null
+++ b/docs/topics/impala_release_notes.xml
@@ -0,0 +1,17 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="impala_release_notes">
+
+  <title>Impala Release Notes</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Release Notes"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody conref="impala_relnotes.xml#relnotes/relnotes_intro"/>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_relnotes.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_relnotes.xml b/docs/topics/impala_relnotes.xml
new file mode 100644
index 0000000..5c53a21
--- /dev/null
+++ b/docs/topics/impala_relnotes.xml
@@ -0,0 +1,34 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="relnotes" audience="standalone">
+
+  <title>Impala Release Notes</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Release Notes"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody id="relnotes_intro">
+
+    <p>
+      These release notes provide information on the <xref href="impala_new_features.xml#new_features">new
+      features</xref> and <xref href="impala_known_issues.xml#known_issues">known issues and limitations</xref> for
+      Impala versions up to <ph conref="../shared/ImpalaVariables.xml#impala_vars/ReleaseVersion"/>. For users
+      upgrading from earlier Impala releases, or using Impala in combination with specific versions of other
+      Cloudera software, <xref href="impala_incompatible_changes.xml#incompatible_changes"/> lists any changes to
+      file formats, SQL syntax, or software dependencies to take into account.
+    </p>
+
+    <p>
+      Once you are finished reviewing these release notes, for more information about using Impala, see
+      <xref audience="integrated" href="impala.xml"/><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/impala.html" scope="external" format="html"/>.
+    </p>
+
+    <p outputclass="toc"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_replica_preference.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_replica_preference.xml b/docs/topics/impala_replica_preference.xml
new file mode 100644
index 0000000..6cf73da
--- /dev/null
+++ b/docs/topics/impala_replica_preference.xml
@@ -0,0 +1,48 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="replica_preference" rev="2.7.0">
+
+  <title>REPLICA_PREFERENCE Query Option (<keyword keyref="impala27"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>REPLICA_PREFERENCE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.7.0">
+      <indexterm audience="Cloudera">REPLICA_PREFERENCE query option</indexterm>
+    </p>
+
+    <p>
+      The <codeph>REPLICA_PREFERENCE</codeph> query option
+      lets you spread the load more evenly if hotspots and bottlenecks persist, by allowing hosts to do local reads,
+      or even remote reads, to retrieve the data for cached blocks if Impala can determine that it would be
+      too expensive to do all such processing on a particular host.
+    </p>
+
+    <p>
+      <b>Type:</b> numeric (0, 3, 5)
+      or corresponding mnemonic strings (<codeph>CACHE_LOCAL</codeph>, <codeph>DISK_LOCAL</codeph>, <codeph>REMOTE</codeph>).
+      The gaps in the numeric sequence are to accomodate other intermediate
+      values that might be added in the future.
+    </p>
+
+    <p>
+      <b>Default:</b> 0 (equivalent to <codeph>CACHE_LOCAL</codeph>)
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/added_in_270"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_perf_hdfs_caching.xml#hdfs_caching"/>, <xref href="impala_schedule_random_replica.xml#schedule_random_replica"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_request_pool.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_request_pool.xml b/docs/topics/impala_request_pool.xml
new file mode 100644
index 0000000..a820edd
--- /dev/null
+++ b/docs/topics/impala_request_pool.xml
@@ -0,0 +1,43 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.3.0" id="request_pool">
+
+  <title>REQUEST_POOL Query Option</title>
+  <titlealts audience="PDF"><navtitle>REQUEST_POOL</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Resource Management"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Admission Control"/>
+      <data name="Category" value="YARN"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">REQUEST_POOL query option</indexterm>
+      The pool or queue name that queries should be submitted to. Only applies when you enable the Impala admission control feature.
+      Specifies the name of the pool used by requests from Impala to the resource manager.
+    </p>
+
+    <p>
+      <b>Type:</b> <codeph>STRING</codeph>
+    </p>
+
+    <p>
+      <b>Default:</b> empty (use the user-to-pool mapping defined by an <cmdname>impalad</cmdname> startup option
+      in the Impala configuration file)
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_admission.xml"/>
+    </p>
+
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_reservation_request_timeout.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_reservation_request_timeout.xml b/docs/topics/impala_reservation_request_timeout.xml
new file mode 100644
index 0000000..0e01f83
--- /dev/null
+++ b/docs/topics/impala_reservation_request_timeout.xml
@@ -0,0 +1,38 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.2" id="reservation_request_timeout">
+
+  <title>RESERVATION_REQUEST_TIMEOUT Query Option (CDH 5 only)</title>
+  <titlealts audience="PDF"><navtitle>RESERVATION_REQUEST_TIMEOUT</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Resource Management"/>
+      <data name="Category" value="YARN"/>
+      <data name="Category" value="Llama"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <note conref="../shared/impala_common.xml#common/llama_query_options_obsolete"/>
+
+    <p>
+      <indexterm audience="Cloudera">RESERVATION_REQUEST_TIMEOUT query option</indexterm>
+      Maximum number of milliseconds Impala will wait for a reservation to be completely granted or denied. Used in
+      conjunction with the Impala resource management feature in Impala 1.2 and higher with CDH 5.
+    </p>
+
+    <p>
+      <b>Type:</b> numeric
+    </p>
+
+    <p>
+      <b>Default:</b> 300000 (5 minutes)
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_reserved_words.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_reserved_words.xml b/docs/topics/impala_reserved_words.xml
new file mode 100644
index 0000000..79dfb5c
--- /dev/null
+++ b/docs/topics/impala_reserved_words.xml
@@ -0,0 +1,365 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="reserved_words">
+
+  <title>Impala Reserved Words</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Planning"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">reserved words</indexterm>
+      The following are the reserved words for the current release of Impala. A reserved word is one that
+      cannot be used directly as an identifier; you must quote it with backticks. For example, a statement
+      <codeph>CREATE TABLE select (x INT)</codeph> fails, while <codeph>CREATE TABLE `select` (x INT)</codeph>
+      succeeds. Impala does not reserve the names of aggregate or scalar built-in functions. (Formerly, Impala did
+      reserve the names of some aggregate functions.)
+    </p>
+
+    <p>
+      Because different database systems have different sets of reserved words, and the reserved words change from
+      release to release, carefully consider database, table, and column names to ensure maximum compatibility
+      between products and versions.
+    </p>
+
+    <p>
+      Because you might switch between Impala and Hive when doing analytics and ETL, also consider whether
+      your object names are the same as any Hive keywords, and rename or quote any that conflict. Consult the
+      <xref href="https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Keywords,Non-reservedKeywordsandReservedKeywords" scope="external" format="html">list of Hive keywords</xref>.
+    </p>
+
+    <p outputclass="toc inpage"/>
+
+  </conbody>
+
+<concept id="reserved_words_current">
+<title>List of Current Reserved Words</title>
+<conbody>
+<!-- This list is derived from the source code at:
+     https://github.com/cloudera/Impala/blob/master/fe/src/main/jflex/sql-scanner.flex
+
+See the history, any recent changes, here:
+     https://github.com/cloudera/Impala/commits/master/fe/src/main/jflex/sql-scanner.flex
+-->
+
+<codeblock rev="ver">add
+aggregate
+all
+alter
+<ph rev="2.0.0">analytic</ph>
+and
+<ph rev="2.0.0">anti</ph>
+<ph rev="1.4.0">api_version</ph>
+as
+asc
+avro
+between
+bigint
+<ph rev="1.4.0">binary</ph>
+boolean
+<ph rev="2.6.0">buckets</ph>
+by
+<ph rev="1.4.0">cached</ph>
+<ph rev="2.3.0">cascade</ph>
+case
+cast
+change
+<ph rev="2.0.0">char</ph>
+<ph rev="1.4.0">class</ph>
+<ph rev="1.2.1">close_fn</ph>
+column
+columns
+comment
+compute
+create
+cross
+<ph rev="2.0.0">current</ph>
+data
+database
+databases
+date
+datetime
+decimal
+<ph rev="2.6.0">delete</ph>
+delimited
+desc
+describe
+distinct
+<ph rev="2.6.0">distribute</ph>
+div
+double
+drop
+else
+end
+escaped
+exists
+explain
+<ph rev="2.5.0">extended</ph>
+external
+false
+fields
+fileformat
+<ph rev="1.2.1">finalize_fn</ph>
+first
+float
+<ph rev="2.0.0">following</ph>
+<ph rev="2.1.0">for</ph>
+format
+formatted
+from
+full
+function
+functions
+<ph rev="2.1.0">grant</ph>
+group
+<ph rev="2.6.0">hash</ph>
+having
+if
+<ph rev="2.6.0">ignore</ph>
+<ph rev="2.5.0">ilike</ph>
+in
+<ph rev="2.1.0">incremental</ph>
+<ph rev="1.2.1">init_fn</ph>
+inner
+inpath
+insert
+int
+integer
+intermediate
+interval
+into
+invalidate
+<ph rev="2.5.0">iregexp</ph>
+is
+join
+last
+left
+like
+limit
+lines
+load
+location
+<ph rev="1.2.1">merge_fn</ph>
+metadata
+not
+null
+nulls
+offset
+on
+or
+order
+outer
+<ph rev="2.0.0">over</ph>
+overwrite
+parquet
+parquetfile
+partition
+partitioned
+<ph rev="1.4.0">partitions</ph>
+<ph rev="2.0.0">preceding</ph>
+<ph rev="1.2.1">prepare_fn</ph>
+<ph rev="1.4.0">produced</ph>
+<ph rev="2.3.0">purge</ph>
+<ph rev="2.0.0">range</ph>
+rcfile
+real
+refresh
+regexp
+rename
+replace
+<ph rev="2.3.0">restrict</ph>
+returns
+<ph rev="2.1.0">revoke</ph>
+right
+rlike
+<ph rev="2.1.0">role</ph>
+<ph rev="2.1.0">roles</ph>
+row
+<ph rev="2.0.0">rows</ph>
+schema
+schemas
+select
+semi
+sequencefile
+serdeproperties
+<ph rev="2.0.0">serialize_fn</ph>
+set
+show
+smallint
+<ph rev="2.6.0">split</ph>
+stats
+stored
+straight_join
+string
+symbol
+table
+tables
+tblproperties
+terminated
+textfile
+then
+timestamp
+tinyint
+to
+true
+<ph rev="2.0.0">truncate</ph>
+<ph rev="2.0.0">unbounded</ph>
+<ph rev="1.4.0">uncached</ph>
+union
+<ph rev="2.6.0">update</ph>
+<ph rev="1.2.1">update_fn</ph>
+use
+using
+values
+<ph rev="2.0.0">varchar</ph>
+view
+when
+where
+with</codeblock>
+</conbody>
+</concept>
+
+<concept id="reserved_words_planning">
+<title>Planning for Future Reserved Words</title>
+<conbody>
+<p>
+The previous list of reserved words includes all the keywords
+used in the current level of Impala SQL syntax.
+To future-proof your code,
+you should avoid additional words in case they
+become reserved words if
+Impala adds features in later releases.
+This kind of planning can also help to avoid
+name conflicts in case you port SQL from other systems that
+have different sets of reserved words.
+</p>
+
+<p>
+The following list contains additional words that Cloudera
+recommends avoiding for table, column, or other object names,
+even though they are not currently reserved by Impala.
+</p>
+
+<codeblock>any
+authorization
+backup
+begin
+break
+browse
+bulk
+cascade
+check
+checkpoint
+close
+clustered
+coalesce
+collate
+commit
+constraint
+contains
+continue
+convert
+current
+current_date
+current_time
+current_timestamp
+current_user
+cursor
+dbcc
+deallocate
+declare
+default
+deny
+disk
+distributed
+dump
+errlvl
+escape
+except
+exec
+execute
+exit
+fetch
+file
+fillfactor
+for
+foreign
+freetext
+goto
+holdlock
+identity
+index
+intersect
+key
+kill
+lineno
+merge
+national
+nocheck
+nonclustered
+nullif
+of
+off
+offsets
+open
+option
+percent
+pivot
+plan
+precision
+primary
+print
+proc
+procedure
+public
+raiserror
+read
+readtext
+reconfigure
+references
+replication
+restore
+restrict
+return
+revert
+rollback
+rowcount
+rule
+save
+securityaudit
+session_user
+setuser
+shutdown
+some
+statistics
+system_user
+tablesample
+textsize
+then
+top
+tran
+transaction
+trigger
+try_convert
+unique
+unpivot
+updatetext
+user
+varying
+waitfor
+while
+within
+writetext
+</codeblock>
+</conbody>
+</concept>
+
+</concept>


[34/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_describe.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_describe.xml b/docs/topics/impala_describe.xml
new file mode 100644
index 0000000..c4070c3
--- /dev/null
+++ b/docs/topics/impala_describe.xml
@@ -0,0 +1,689 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="describe">
+
+  <title id="desc">DESCRIBE Statement</title>
+  <titlealts audience="PDF"><navtitle>DESCRIBE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Reports"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">DESCRIBE statement</indexterm>
+      The <codeph>DESCRIBE</codeph> statement displays metadata about a table, such as the column names and their
+      data types.
+      <ph rev="2.3.0">In <keyword keyref="impala23_full"/> and higher, you can specify the name of a complex type column, which takes
+      the form of a dotted path. The path might include multiple components in the case of a nested type definition.</ph>
+      <ph rev="2.5.0">In <keyword keyref="impala25_full"/> and higher, the <codeph>DESCRIBE DATABASE</codeph> form can display
+      information about a database.</ph>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock rev="2.5.0">DESCRIBE [DATABASE] [FORMATTED|EXTENDED] <varname>object_name</varname>
+
+object_name ::=
+    [<varname>db_name</varname>.]<varname>table_name</varname>[.<varname>complex_col_name</varname> ...]
+  | <varname>db_name</varname>
+</codeblock>
+
+    <p>
+      You can use the abbreviation <codeph>DESC</codeph> for the <codeph>DESCRIBE</codeph> statement.
+    </p>
+
+    <p rev="1.1">
+      The <codeph>DESCRIBE FORMATTED</codeph> variation displays additional information, in a format familiar to
+      users of Apache Hive. The extra information includes low-level details such as whether the table is internal
+      or external, when it was created, the file format, the location of the data in HDFS, whether the object is a
+      table or a view, and (for views) the text of the query from the view definition.
+    </p>
+
+    <note>
+      The <codeph>Compressed</codeph> field is not a reliable indicator of whether the table contains compressed
+      data. It typically always shows <codeph>No</codeph>, because the compression settings only apply during the
+      session that loads data and are not stored persistently with the table metadata.
+    </note>
+
+<p rev="2.5.0 IMPALA-2196">
+  <b>Describing databases:</b>
+</p>
+
+<p rev="2.5.0">
+  By default, the <codeph>DESCRIBE</codeph> output for a database includes the location
+  and the comment, which can be set by the <codeph>LOCATION</codeph> and <codeph>COMMENT</codeph>
+  clauses on the <codeph>CREATE DATABASE</codeph> statement.
+</p>
+
+<p rev="2.5.0">
+  The additional information displayed by the <codeph>FORMATTED</codeph> or <codeph>EXTENDED</codeph>
+  keyword includes the HDFS user ID that is considered the owner of the database, and any
+  optional database properties. The properties could be specified by the <codeph>WITH DBPROPERTIES</codeph>
+  clause if the database is created using a Hive <codeph>CREATE DATABASE</codeph> statement.
+  Impala currently does not set or do any special processing based on those properties.
+</p>
+
+<p rev="2.5.0">
+The following examples show the variations in syntax and output for
+describing databases. This feature is available in <keyword keyref="impala25_full"/>
+and higher.
+</p>
+
+<codeblock rev="2.5.0">
+describe database default;
++---------+----------------------+-----------------------+
+| name    | location             | comment               |
++---------+----------------------+-----------------------+
+| default | /user/hive/warehouse | Default Hive database |
++---------+----------------------+-----------------------+
+
+describe database formatted default;
++---------+----------------------+-----------------------+
+| name    | location             | comment               |
++---------+----------------------+-----------------------+
+| default | /user/hive/warehouse | Default Hive database |
+| Owner:  |                      |                       |
+|         | public               | ROLE                  |
++---------+----------------------+-----------------------+
+
+describe database extended default;
++---------+----------------------+-----------------------+
+| name    | location             | comment               |
++---------+----------------------+-----------------------+
+| default | /user/hive/warehouse | Default Hive database |
+| Owner:  |                      |                       |
+|         | public               | ROLE                  |
++---------+----------------------+-----------------------+
+</codeblock>
+
+<p>
+  <b>Describing tables:</b>
+</p>
+
+<p>
+  If the <codeph>DATABASE</codeph> keyword is omitted, the default
+  for the <codeph>DESCRIBE</codeph> statement is to refer to a table.
+</p>
+
+<codeblock>
+-- By default, the table is assumed to be in the current database.
+describe my_table;
++------+--------+---------+
+| name | type   | comment |
++------+--------+---------+
+| x    | int    |         |
+| s    | string |         |
++------+--------+---------+
+
+-- Use a fully qualified table name to specify a table in any database.
+describe my_database.my_table;
++------+--------+---------+
+| name | type   | comment |
++------+--------+---------+
+| x    | int    |         |
+| s    | string |         |
++------+--------+---------+
+
+-- The formatted or extended output includes additional useful information.
+-- The LOCATION field is especially useful to know for DDL statements and HDFS commands
+-- during ETL jobs. (The LOCATION includes a full hdfs:// URL, omitted here for readability.)
+describe formatted my_table;
++------------------------------+----------------------------------------------+----------------------+
+| name                         | type                                         | comment              |
++------------------------------+----------------------------------------------+----------------------+
+| # col_name                   | data_type                                    | comment              |
+|                              | NULL                                         | NULL                 |
+| x                            | int                                          | NULL                 |
+| s                            | string                                       | NULL                 |
+|                              | NULL                                         | NULL                 |
+| # Detailed Table Information | NULL                                         | NULL                 |
+| Database:                    | my_database                                  | NULL                 |
+| Owner:                       | jrussell                                     | NULL                 |
+| CreateTime:                  | Fri Mar 18 15:58:00 PDT 2016                 | NULL                 |
+| LastAccessTime:              | UNKNOWN                                      | NULL                 |
+| Protect Mode:                | None                                         | NULL                 |
+| Retention:                   | 0                                            | NULL                 |
+| Location:                    | /user/hive/warehouse/my_database.db/my_table | NULL                 |
+| Table Type:                  | MANAGED_TABLE                                | NULL                 |
+| Table Parameters:            | NULL                                         | NULL                 |
+|                              | transient_lastDdlTime                        | 1458341880           |
+|                              | NULL                                         | NULL                 |
+| # Storage Information        | NULL                                         | NULL                 |
+| SerDe Library:               | org. ... .LazySimpleSerDe                    | NULL                 |
+| InputFormat:                 | org.apache.hadoop.mapred.TextInputFormat     | NULL                 |
+| OutputFormat:                | org. ... .HiveIgnoreKeyTextOutputFormat      | NULL                 |
+| Compressed:                  | No                                           | NULL                 |
+| Num Buckets:                 | 0                                            | NULL                 |
+| Bucket Columns:              | []                                           | NULL                 |
+| Sort Columns:                | []                                           | NULL                 |
++------------------------------+----------------------------------------------+----------------------+
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+    <p rev="2.3.0">
+      Because the column definitions for complex types can become long, particularly when such types are nested,
+      the <codeph>DESCRIBE</codeph> statement uses special formatting for complex type columns to make the output readable.
+    </p>
+
+    <p rev="2.3.0">
+      For the <codeph>ARRAY</codeph>, <codeph>STRUCT</codeph>, and <codeph>MAP</codeph> types available in
+      <keyword keyref="impala23_full"/> and higher, the <codeph>DESCRIBE</codeph> output is formatted to avoid
+      excessively long lines for multiple fields within a <codeph>STRUCT</codeph>, or a nested sequence of
+      complex types.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_describe"/>
+
+    <p rev="2.3.0">
+      For example, here is the <codeph>DESCRIBE</codeph> output for a table containing a single top-level column
+      of each complex type:
+    </p>
+
+<codeblock rev="2.3.0"><![CDATA[create table t1 (x int, a array<int>, s struct<f1: string, f2: bigint>, m map<string,int>) stored as parquet;
+
+describe t1;
++------+-----------------+---------+
+| name | type            | comment |
++------+-----------------+---------+
+| x    | int             |         |
+| a    | array<int>      |         |
+| s    | struct<         |         |
+|      |   f1:string,    |         |
+|      |   f2:bigint     |         |
+|      | >               |         |
+| m    | map<string,int> |         |
++------+-----------------+---------+
+]]>
+</codeblock>
+
+    <p rev="2.3.0">
+      Here are examples showing how to <q>drill down</q> into the layouts of complex types, including
+      using multi-part names to examine the definitions of nested types.
+      The <codeph>&lt; &gt;</codeph> delimiters identify the columns with complex types;
+      these are the columns where you can descend another level to see the parts that make up
+      the complex type.
+      This technique helps you to understand the multi-part names you use as table references in queries
+      involving complex types, and the corresponding column names you refer to in the <codeph>SELECT</codeph> list.
+      These tables are from the <q>nested TPC-H</q> schema, shown in detail in
+      <xref href="impala_complex_types.xml#complex_sample_schema"/>.
+    </p>
+
+    <p>
+      The <codeph>REGION</codeph> table contains an <codeph>ARRAY</codeph> of <codeph>STRUCT</codeph>
+      elements:
+    </p>
+
+    <ul>
+      <li>
+        <p>
+          The first <codeph>DESCRIBE</codeph> specifies the table name, to display the definition
+          of each top-level column.
+        </p>
+      </li>
+      <li>
+        <p>
+          The second <codeph>DESCRIBE</codeph> specifies the name of a complex
+          column, <codeph>REGION.R_NATIONS</codeph>, showing that when you include the name of an <codeph>ARRAY</codeph>
+          column in a <codeph>FROM</codeph> clause, that table reference acts like a two-column table with
+          columns <codeph>ITEM</codeph> and <codeph>POS</codeph>.
+        </p>
+      </li>
+      <li>
+        <p>
+          The final <codeph>DESCRIBE</codeph> specifies the fully qualified name of the <codeph>ITEM</codeph> field,
+          to display the layout of its underlying <codeph>STRUCT</codeph> type in table format, with the fields
+          mapped to column names.
+        </p>
+      </li>
+    </ul>
+
+<codeblock rev="2.3.0"><![CDATA[
+-- #1: The overall layout of the entire table.
+describe region;
++-------------+-------------------------+---------+
+| name        | type                    | comment |
++-------------+-------------------------+---------+
+| r_regionkey | smallint                |         |
+| r_name      | string                  |         |
+| r_comment   | string                  |         |
+| r_nations   | array<struct<           |         |
+|             |   n_nationkey:smallint, |         |
+|             |   n_name:string,        |         |
+|             |   n_comment:string      |         |
+|             | >>                      |         |
++-------------+-------------------------+---------+
+
+-- #2: The ARRAY column within the table.
+describe region.r_nations;
++------+-------------------------+---------+
+| name | type                    | comment |
++------+-------------------------+---------+
+| item | struct<                 |         |
+|      |   n_nationkey:smallint, |         |
+|      |   n_name:string,        |         |
+|      |   n_comment:string      |         |
+|      | >                       |         |
+| pos  | bigint                  |         |
++------+-------------------------+---------+
+
+-- #3: The STRUCT that makes up each ARRAY element.
+--     The fields of the STRUCT act like columns of a table.
+describe region.r_nations.item;
++-------------+----------+---------+
+| name        | type     | comment |
++-------------+----------+---------+
+| n_nationkey | smallint |         |
+| n_name      | string   |         |
+| n_comment   | string   |         |
++-------------+----------+---------+
+]]>
+</codeblock>
+
+    <p>
+      The <codeph>CUSTOMER</codeph> table contains an <codeph>ARRAY</codeph> of <codeph>STRUCT</codeph>
+      elements, where one field in the <codeph>STRUCT</codeph> is another <codeph>ARRAY</codeph> of
+      <codeph>STRUCT</codeph> elements:
+    </p>
+    <ul>
+      <li>
+        <p>
+          Again, the initial <codeph>DESCRIBE</codeph> specifies only the table name.
+        </p>
+      </li>
+      <li>
+        <p>
+          The second <codeph>DESCRIBE</codeph> specifies the qualified name of the complex
+          column, <codeph>CUSTOMER.C_ORDERS</codeph>, showing how an <codeph>ARRAY</codeph>
+          is represented as a two-column table with columns <codeph>ITEM</codeph> and <codeph>POS</codeph>.
+        </p>
+      </li>
+      <li>
+        <p>
+          The third <codeph>DESCRIBE</codeph> specifies the qualified name of the <codeph>ITEM</codeph>
+          of the <codeph>ARRAY</codeph> column, to see the structure of the nested <codeph>ARRAY</codeph>.
+          Again, it has has two parts, <codeph>ITEM</codeph> and <codeph>POS</codeph>. Because the
+          <codeph>ARRAY</codeph> contains a <codeph>STRUCT</codeph>, the layout of the <codeph>STRUCT</codeph>
+          is shown.
+        </p>
+      </li>
+      <li>
+        <p>
+          The fourth and fifth <codeph>DESCRIBE</codeph> statements drill down into a <codeph>STRUCT</codeph> field that
+          is itself a complex type, an <codeph>ARRAY</codeph> of <codeph>STRUCT</codeph>.
+          The <codeph>ITEM</codeph> portion of the qualified name is only required when the <codeph>ARRAY</codeph>
+          elements are anonymous. The fields of the <codeph>STRUCT</codeph> give names to any other complex types
+          nested inside the <codeph>STRUCT</codeph>. Therefore, the <codeph>DESCRIBE</codeph> parameters
+          <codeph>CUSTOMER.C_ORDERS.ITEM.O_LINEITEMS</codeph> and <codeph>CUSTOMER.C_ORDERS.O_LINEITEMS</codeph>
+          are equivalent. (For brevity, leave out the <codeph>ITEM</codeph> portion of
+          a qualified name when it is not required.)
+        </p>
+      </li>
+      <li>
+        <p>
+          The final <codeph>DESCRIBE</codeph> shows the layout of the deeply nested <codeph>STRUCT</codeph> type.
+          Because there are no more complex types nested inside this <codeph>STRUCT</codeph>, this is as far
+          as you can drill down into the layout for this table.
+        </p>
+      </li>
+    </ul>
+
+<codeblock rev="2.3.0"><![CDATA[-- #1: The overall layout of the entire table.
+describe customer;
++--------------+------------------------------------+
+| name         | type                               |
++--------------+------------------------------------+
+| c_custkey    | bigint                             |
+... more scalar columns ...
+| c_orders     | array<struct<                      |
+|              |   o_orderkey:bigint,               |
+|              |   o_orderstatus:string,            |
+|              |   o_totalprice:decimal(12,2),      |
+|              |   o_orderdate:string,              |
+|              |   o_orderpriority:string,          |
+|              |   o_clerk:string,                  |
+|              |   o_shippriority:int,              |
+|              |   o_comment:string,                |
+|              |   o_lineitems:array<struct<        |
+|              |     l_partkey:bigint,              |
+|              |     l_suppkey:bigint,              |
+|              |     l_linenumber:int,              |
+|              |     l_quantity:decimal(12,2),      |
+|              |     l_extendedprice:decimal(12,2), |
+|              |     l_discount:decimal(12,2),      |
+|              |     l_tax:decimal(12,2),           |
+|              |     l_returnflag:string,           |
+|              |     l_linestatus:string,           |
+|              |     l_shipdate:string,             |
+|              |     l_commitdate:string,           |
+|              |     l_receiptdate:string,          |
+|              |     l_shipinstruct:string,         |
+|              |     l_shipmode:string,             |
+|              |     l_comment:string               |
+|              |   >>                               |
+|              | >>                                 |
++--------------+------------------------------------+
+
+-- #2: The ARRAY column within the table.
+describe customer.c_orders;
++------+------------------------------------+
+| name | type                               |
++------+------------------------------------+
+| item | struct<                            |
+|      |   o_orderkey:bigint,               |
+|      |   o_orderstatus:string,            |
+... more struct fields ...
+|      |   o_lineitems:array<struct<        |
+|      |     l_partkey:bigint,              |
+|      |     l_suppkey:bigint,              |
+... more nested struct fields ...
+|      |     l_comment:string               |
+|      |   >>                               |
+|      | >                                  |
+| pos  | bigint                             |
++------+------------------------------------+
+
+-- #3: The STRUCT that makes up each ARRAY element.
+--     The fields of the STRUCT act like columns of a table.
+describe customer.c_orders.item;
++-----------------+----------------------------------+
+| name            | type                             |
++-----------------+----------------------------------+
+| o_orderkey      | bigint                           |
+| o_orderstatus   | string                           |
+| o_totalprice    | decimal(12,2)                    |
+| o_orderdate     | string                           |
+| o_orderpriority | string                           |
+| o_clerk         | string                           |
+| o_shippriority  | int                              |
+| o_comment       | string                           |
+| o_lineitems     | array<struct<                    |
+|                 |   l_partkey:bigint,              |
+|                 |   l_suppkey:bigint,              |
+... more struct fields ...
+|                 |   l_comment:string               |
+|                 | >>                               |
++-----------------+----------------------------------+
+
+-- #4: The ARRAY nested inside the STRUCT elements of the first ARRAY.
+describe customer.c_orders.item.o_lineitems;
++------+----------------------------------+
+| name | type                             |
++------+----------------------------------+
+| item | struct<                          |
+|      |   l_partkey:bigint,              |
+|      |   l_suppkey:bigint,              |
+... more struct fields ...
+|      |   l_comment:string               |
+|      | >                                |
+| pos  | bigint                           |
++------+----------------------------------+
+
+-- #5: Shorter form of the previous DESCRIBE. Omits the .ITEM portion of the name
+--     because O_LINEITEMS and other field names provide a way to refer to things
+--     inside the ARRAY element.
+describe customer.c_orders.o_lineitems;
++------+----------------------------------+
+| name | type                             |
++------+----------------------------------+
+| item | struct<                          |
+|      |   l_partkey:bigint,              |
+|      |   l_suppkey:bigint,              |
+... more struct fields ...
+|      |   l_comment:string               |
+|      | >                                |
+| pos  | bigint                           |
++------+----------------------------------+
+
+-- #6: The STRUCT representing ARRAY elements nested inside
+--     another ARRAY of STRUCTs. The lack of any complex types
+--     in this output means this is as far as DESCRIBE can
+--     descend into the table layout.
+describe customer.c_orders.o_lineitems.item;
++-----------------+---------------+
+| name            | type          |
++-----------------+---------------+
+| l_partkey       | bigint        |
+| l_suppkey       | bigint        |
+... more scalar columns ...
+| l_comment       | string        |
++-----------------+---------------+
+]]>
+</codeblock>
+
+<p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+<p>
+  After the <cmdname>impalad</cmdname> daemons are restarted, the first query against a table can take longer
+  than subsequent queries, because the metadata for the table is loaded before the query is processed. This
+  one-time delay for each table can cause misleading results in benchmark tests or cause unnecessary concern.
+  To <q>warm up</q> the Impala metadata cache, you can issue a <codeph>DESCRIBE</codeph> statement in advance
+  for each table you intend to access later.
+</p>
+
+<p>
+  When you are dealing with data files stored in HDFS, sometimes it is important to know details such as the
+  path of the data files for an Impala table, and the hostname for the namenode. You can get this information
+  from the <codeph>DESCRIBE FORMATTED</codeph> output. You specify HDFS URIs or path specifications with
+  statements such as <codeph>LOAD DATA</codeph> and the <codeph>LOCATION</codeph> clause of <codeph>CREATE
+  TABLE</codeph> or <codeph>ALTER TABLE</codeph>. You might also use HDFS URIs or paths with Linux commands
+  such as <cmdname>hadoop</cmdname> and <cmdname>hdfs</cmdname> to copy, rename, and so on, data files in HDFS.
+</p>
+
+<p conref="../shared/impala_common.xml#common/sync_ddl_blurb"/>
+
+<p rev="1.2.1">
+  Each table can also have associated table statistics and column statistics. To see these categories of
+  information, use the <codeph>SHOW TABLE STATS <varname>table_name</varname></codeph> and <codeph>SHOW COLUMN
+  STATS <varname>table_name</varname></codeph> statements.
+<!--
+For example, the table statistics can often show you the number
+and total size of the files in the table, even if you have not
+run <codeph>COMPUTE STATS</codeph>.
+-->
+  See <xref href="impala_show.xml#show"/> for details.
+</p>
+
+<note conref="../shared/impala_common.xml#common/compute_stats_next"/>
+
+<p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<p>
+  The following example shows the results of both a standard <codeph>DESCRIBE</codeph> and <codeph>DESCRIBE
+  FORMATTED</codeph> for different kinds of schema objects:
+</p>
+
+  <ul>
+    <li>
+      <codeph>DESCRIBE</codeph> for a table or a view returns the name, type, and comment for each of the
+      columns. For a view, if the column value is computed by an expression, the column name is automatically
+      generated as <codeph>_c0</codeph>, <codeph>_c1</codeph>, and so on depending on the ordinal number of the
+      column.
+    </li>
+
+    <li>
+      A table created with no special format or storage clauses is designated as a <codeph>MANAGED_TABLE</codeph>
+      (an <q>internal table</q> in Impala terminology). Its data files are stored in an HDFS directory under the
+      default Hive data directory. By default, it uses Text data format.
+    </li>
+
+    <li>
+      A view is designated as <codeph>VIRTUAL_VIEW</codeph> in <codeph>DESCRIBE FORMATTED</codeph> output. Some
+      of its properties are <codeph>NULL</codeph> or blank because they are inherited from the base table. The
+      text of the query that defines the view is part of the <codeph>DESCRIBE FORMATTED</codeph> output.
+    </li>
+
+    <li>
+      A table with additional clauses in the <codeph>CREATE TABLE</codeph> statement has differences in
+      <codeph>DESCRIBE FORMATTED</codeph> output. The output for <codeph>T2</codeph> includes the
+      <codeph>EXTERNAL_TABLE</codeph> keyword because of the <codeph>CREATE EXTERNAL TABLE</codeph> syntax, and
+      different <codeph>InputFormat</codeph> and <codeph>OutputFormat</codeph> fields to reflect the Parquet file
+      format.
+    </li>
+  </ul>
+
+<codeblock>[localhost:21000] &gt; create table t1 (x int, y int, s string);
+Query: create table t1 (x int, y int, s string)
+[localhost:21000] &gt; describe t1;
+Query: describe t1
+Query finished, fetching results ...
++------+--------+---------+
+| name | type   | comment |
++------+--------+---------+
+| x    | int    |         |
+| y    | int    |         |
+| s    | string |         |
++------+--------+---------+
+Returned 3 row(s) in 0.13s
+[localhost:21000] &gt; describe formatted t1;
+Query: describe formatted t1
+Query finished, fetching results ...
++------------------------------+--------------------------------------------+------------+
+| name                         | type                                       | comment    |
++------------------------------+--------------------------------------------+------------+
+| # col_name                   | data_type                                  | comment    |
+|                              | NULL                                       | NULL       |
+| x                            | int                                        | None       |
+| y                            | int                                        | None       |
+| s                            | string                                     | None       |
+|                              | NULL                                       | NULL       |
+| # Detailed Table Information | NULL                                       | NULL       |
+| Database:                    | describe_formatted                         | NULL       |
+| Owner:                       | cloudera                                   | NULL       |
+| CreateTime:                  | Mon Jul 22 17:03:16 EDT 2013               | NULL       |
+| LastAccessTime:              | UNKNOWN                                    | NULL       |
+| Protect Mode:                | None                                       | NULL       |
+| Retention:                   | 0                                          | NULL       |
+| Location:                    | hdfs://127.0.0.1:8020/user/hive/warehouse/ |            |
+|                              |   describe_formatted.db/t1                 | NULL       |
+| Table Type:                  | MANAGED_TABLE                              | NULL       |
+| Table Parameters:            | NULL                                       | NULL       |
+|                              | transient_lastDdlTime                      | 1374526996 |
+|                              | NULL                                       | NULL       |
+| # Storage Information        | NULL                                       | NULL       |
+| SerDe Library:               | org.apache.hadoop.hive.serde2.lazy.        |            |
+|                              |   LazySimpleSerDe                          | NULL       |
+| InputFormat:                 | org.apache.hadoop.mapred.TextInputFormat   | NULL       |
+| OutputFormat:                | org.apache.hadoop.hive.ql.io.              |            |
+|                              |   HiveIgnoreKeyTextOutputFormat            | NULL       |
+| Compressed:                  | No                                         | NULL       |
+| Num Buckets:                 | 0                                          | NULL       |
+| Bucket Columns:              | []                                         | NULL       |
+| Sort Columns:                | []                                         | NULL       |
++------------------------------+--------------------------------------------+------------+
+Returned 26 row(s) in 0.03s
+[localhost:21000] &gt; create view v1 as select x, upper(s) from t1;
+Query: create view v1 as select x, upper(s) from t1
+[localhost:21000] &gt; describe v1;
+Query: describe v1
+Query finished, fetching results ...
++------+--------+---------+
+| name | type   | comment |
++------+--------+---------+
+| x    | int    |         |
+| _c1  | string |         |
++------+--------+---------+
+Returned 2 row(s) in 0.10s
+[localhost:21000] &gt; describe formatted v1;
+Query: describe formatted v1
+Query finished, fetching results ...
++------------------------------+------------------------------+----------------------+
+| name                         | type                         | comment              |
++------------------------------+------------------------------+----------------------+
+| # col_name                   | data_type                    | comment              |
+|                              | NULL                         | NULL                 |
+| x                            | int                          | None                 |
+| _c1                          | string                       | None                 |
+|                              | NULL                         | NULL                 |
+| # Detailed Table Information | NULL                         | NULL                 |
+| Database:                    | describe_formatted           | NULL                 |
+| Owner:                       | cloudera                     | NULL                 |
+| CreateTime:                  | Mon Jul 22 16:56:38 EDT 2013 | NULL                 |
+| LastAccessTime:              | UNKNOWN                      | NULL                 |
+| Protect Mode:                | None                         | NULL                 |
+| Retention:                   | 0                            | NULL                 |
+| Table Type:                  | VIRTUAL_VIEW                 | NULL                 |
+| Table Parameters:            | NULL                         | NULL                 |
+|                              | transient_lastDdlTime        | 1374526598           |
+|                              | NULL                         | NULL                 |
+| # Storage Information        | NULL                         | NULL                 |
+| SerDe Library:               | null                         | NULL                 |
+| InputFormat:                 | null                         | NULL                 |
+| OutputFormat:                | null                         | NULL                 |
+| Compressed:                  | No                           | NULL                 |
+| Num Buckets:                 | 0                            | NULL                 |
+| Bucket Columns:              | []                           | NULL                 |
+| Sort Columns:                | []                           | NULL                 |
+|                              | NULL                         | NULL                 |
+| # View Information           | NULL                         | NULL                 |
+| View Original Text:          | SELECT x, upper(s) FROM t1   | NULL                 |
+| View Expanded Text:          | SELECT x, upper(s) FROM t1   | NULL                 |
++------------------------------+------------------------------+----------------------+
+Returned 28 row(s) in 0.03s
+[localhost:21000] &gt; create external table t2 (x int, y int, s string) stored as parquet location '/user/cloudera/sample_data';
+[localhost:21000] &gt; describe formatted t2;
+Query: describe formatted t2
+Query finished, fetching results ...
++------------------------------+----------------------------------------------------+------------+
+| name                         | type                                               | comment    |
++------------------------------+----------------------------------------------------+------------+
+| # col_name                   | data_type                                          | comment    |
+|                              | NULL                                               | NULL       |
+| x                            | int                                                | None       |
+| y                            | int                                                | None       |
+| s                            | string                                             | None       |
+|                              | NULL                                               | NULL       |
+| # Detailed Table Information | NULL                                               | NULL       |
+| Database:                    | describe_formatted                                 | NULL       |
+| Owner:                       | cloudera                                           | NULL       |
+| CreateTime:                  | Mon Jul 22 17:01:47 EDT 2013                       | NULL       |
+| LastAccessTime:              | UNKNOWN                                            | NULL       |
+| Protect Mode:                | None                                               | NULL       |
+| Retention:                   | 0                                                  | NULL       |
+| Location:                    | hdfs://127.0.0.1:8020/user/cloudera/sample_data    | NULL       |
+| Table Type:                  | EXTERNAL_TABLE                                     | NULL       |
+| Table Parameters:            | NULL                                               | NULL       |
+|                              | EXTERNAL                                           | TRUE       |
+|                              | transient_lastDdlTime                              | 1374526907 |
+|                              | NULL                                               | NULL       |
+| # Storage Information        | NULL                                               | NULL       |
+| SerDe Library:               | org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | NULL       |
+| InputFormat:                 | com.cloudera.impala.hive.serde.ParquetInputFormat  | NULL       |
+| OutputFormat:                | com.cloudera.impala.hive.serde.ParquetOutputFormat | NULL       |
+| Compressed:                  | No                                                 | NULL       |
+| Num Buckets:                 | 0                                                  | NULL       |
+| Bucket Columns:              | []                                                 | NULL       |
+| Sort Columns:                | []                                                 | NULL       |
++------------------------------+----------------------------------------------------+------------+
+Returned 27 row(s) in 0.17s</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+    <p rev="CDH-19187">
+      The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+      typically the <codeph>impala</codeph> user, must have read and execute
+      permissions for all directories that are part of the table.
+      (A table could span multiple different HDFS directories if it is partitioned.
+      The directories could be widely scattered because a partition can reside
+      in an arbitrary HDFS directory based on its <codeph>LOCATION</codeph> attribute.)
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_tables.xml#tables"/>, <xref href="impala_create_table.xml#create_table"/>,
+      <xref href="impala_show.xml#show_tables"/>, <xref href="impala_show.xml#show_create_table"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_development.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_development.xml b/docs/topics/impala_development.xml
new file mode 100644
index 0000000..1b35046
--- /dev/null
+++ b/docs/topics/impala_development.xml
@@ -0,0 +1,231 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="intro_dev">
+
+  <title>Developing Impala Applications</title>
+  <titlealts audience="PDF"><navtitle>Developing Applications</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Concepts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      The core development language with Impala is SQL. You can also use Java or other languages to interact with
+      Impala through the standard JDBC and ODBC interfaces used by many business intelligence tools. For
+      specialized kinds of analysis, you can supplement the SQL built-in functions by writing
+      <xref href="impala_udf.xml#udfs">user-defined functions (UDFs)</xref> in C++ or Java.
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="intro_sql">
+
+    <title>Overview of the Impala SQL Dialect</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Concepts"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        The Impala SQL dialect is highly compatible with the SQL syntax used in the Apache Hive component (HiveQL). As
+        such, it is familiar to users who are already familiar with running SQL queries on the Hadoop
+        infrastructure. Currently, Impala SQL supports a subset of HiveQL statements, data types, and built-in
+        functions. Impala also includes additional built-in functions for common industry features, to simplify
+        porting SQL from non-Hadoop systems.
+      </p>
+
+      <p>
+        For users coming to Impala from traditional database or data warehousing backgrounds, the following aspects of the SQL dialect
+        might seem familiar:
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            The <xref href="impala_select.xml#select">SELECT statement</xref> includes familiar clauses such as <codeph>WHERE</codeph>,
+            <codeph>GROUP BY</codeph>, <codeph>ORDER BY</codeph>, and <codeph>WITH</codeph>.
+            You will find familiar notions such as
+            <xref href="impala_joins.xml#joins">joins</xref>, <xref href="impala_functions.xml#builtins">built-in
+            functions</xref> for processing strings, numbers, and dates,
+            <xref href="impala_aggregate_functions.xml#aggregate_functions">aggregate functions</xref>,
+            <xref href="impala_subqueries.xml#subqueries">subqueries</xref>, and
+            <xref href="impala_operators.xml#comparison_operators">comparison operators</xref>
+            such as <codeph>IN()</codeph> and <codeph>BETWEEN</codeph>.
+            The <codeph>SELECT</codeph> statement is the place where SQL standards compliance is most important.
+          </p>
+        </li>
+
+        <li>
+          <p>
+          From the data warehousing world, you will recognize the notion of
+          <xref href="impala_partitioning.xml#partitioning">partitioned tables</xref>.
+          One or more columns serve as partition keys, and the data is physically arranged so that
+          queries that refer to the partition key columns in the <codeph>WHERE</codeph> clause
+          can skip partitions that do not match the filter conditions. For example, if you have 10
+          years worth of data and use a clause such as <codeph>WHERE year = 2015</codeph>,
+          <codeph>WHERE year &gt; 2010</codeph>, or <codeph>WHERE year IN (2014, 2015)</codeph>,
+          Impala skips all the data for non-matching years, greatly reducing the amount of I/O
+          for the query.
+          </p>
+        </li>
+
+        <li rev="1.2">
+          <p>
+          In Impala 1.2 and higher, <xref href="impala_udf.xml#udfs">UDFs</xref> let you perform custom comparisons
+          and transformation logic during <codeph>SELECT</codeph> and <codeph>INSERT...SELECT</codeph> statements.
+          </p>
+        </li>
+      </ul>
+
+      <p>
+        For users coming to Impala from traditional database or data warehousing backgrounds, the following aspects of the SQL dialect
+        might require some learning and practice for you to become proficient in the Hadoop environment:
+      </p>
+
+      <ul>
+        <li>
+          <p>
+          Impala SQL is focused on queries and includes relatively little DML. There is no <codeph>UPDATE</codeph>
+          or <codeph>DELETE</codeph> statement. Stale data is typically discarded (by <codeph>DROP TABLE</codeph>
+          or <codeph>ALTER TABLE ... DROP PARTITION</codeph> statements) or replaced (by <codeph>INSERT
+          OVERWRITE</codeph> statements).
+          </p>
+        </li>
+
+        <li>
+          <p>
+          All data creation is done by <codeph>INSERT</codeph> statements, which typically insert data in bulk by
+          querying from other tables. There are two variations, <codeph>INSERT INTO</codeph> which appends to the
+          existing data, and <codeph>INSERT OVERWRITE</codeph> which replaces the entire contents of a table or
+          partition (similar to <codeph>TRUNCATE TABLE</codeph> followed by a new <codeph>INSERT</codeph>).
+          Although there is an <codeph>INSERT ... VALUES</codeph> syntax to create a small number of values in
+          a single statement, it is far more efficient to use the <codeph>INSERT ... SELECT</codeph> to copy
+          and transform large amounts of data from one table to another in a single operation.
+          </p>
+        </li>
+
+        <li>
+          <p>
+          You often construct Impala table definitions and data files in some other environment, and then attach
+          Impala so that it can run real-time queries. The same data files and table metadata are shared with other
+          components of the Hadoop ecosystem. In particular, Impala can access tables created by Hive or data
+          inserted by Hive, and Hive can access tables and data produced by Impala. Many other Hadoop components
+          can write files in formats such as Parquet and Avro, that can then be queried by Impala.
+          </p>
+        </li>
+
+        <li>
+          <p>
+          Because Hadoop and Impala are focused on data warehouse-style operations on large data sets, Impala SQL
+          includes some idioms that you might find in the import utilities for traditional database systems. For
+          example, you can create a table that reads comma-separated or tab-separated text files, specifying the
+          separator in the <codeph>CREATE TABLE</codeph> statement. You can create <b>external tables</b> that read
+          existing data files but do not move or transform them.
+          </p>
+        </li>
+
+        <li>
+          <p>
+          Because Impala reads large quantities of data that might not be perfectly tidy and predictable, it does
+          not require length constraints on string data types. For example, you can define a database column as
+          <codeph>STRING</codeph> with unlimited length, rather than <codeph>CHAR(1)</codeph> or
+          <codeph>VARCHAR(64)</codeph>. <ph rev="2.0.0">(Although in Impala 2.0 and later, you can also use
+          length-constrained <codeph>CHAR</codeph> and <codeph>VARCHAR</codeph> types.)</ph>
+          </p>
+        </li>
+
+      </ul>
+
+      <p>
+        <b>Related information:</b> <xref href="impala_langref.xml#langref"/>, especially
+        <xref href="impala_langref_sql.xml#langref_sql"/> and <xref href="impala_functions.xml#builtins"/>
+      </p>
+    </conbody>
+  </concept>
+
+<!-- Bunch of potential concept topics for future consideration. Major areas of Impala modelled on areas of discussion for Oracle Database, and distributed databases in general. -->
+
+  <concept id="intro_datatypes" audience="Cloudera">
+
+    <title>Overview of Impala SQL Data Types</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="intro_network" audience="Cloudera">
+
+    <title>Overview of Impala Network Topology</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="intro_cluster" audience="Cloudera">
+
+    <title>Overview of Impala Cluster Topology</title>
+
+    <conbody/>
+  </concept>
+
+  <concept id="intro_apis">
+
+    <title>Overview of Impala Programming Interfaces</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="JDBC"/>
+      <data name="Category" value="ODBC"/>
+      <data name="Category" value="Hue"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        You can connect and submit requests to the Impala daemons through:
+      </p>
+
+      <ul>
+        <li>
+          The <codeph><xref href="impala_impala_shell.xml#impala_shell">impala-shell</xref></codeph> interactive
+          command interpreter.
+        </li>
+
+        <li>
+          The <xref href="http://gethue.com/" scope="external" format="html">Hue</xref> web-based user interface.
+        </li>
+
+        <li>
+          <xref href="impala_jdbc.xml#impala_jdbc">JDBC</xref>.
+        </li>
+
+        <li>
+          <xref href="impala_odbc.xml#impala_odbc">ODBC</xref>.
+        </li>
+      </ul>
+
+      <p>
+        With these options, you can use Impala in heterogeneous environments, with JDBC or ODBC applications
+        running on non-Linux platforms. You can also use Impala on combination with various Business Intelligence
+        tools that use the JDBC and ODBC interfaces.
+      </p>
+
+      <p>
+        Each <codeph>impalad</codeph> daemon process, running on separate nodes in a cluster, listens to
+        <xref href="impala_ports.xml#ports">several ports</xref> for incoming requests. Requests from
+        <codeph>impala-shell</codeph> and Hue are routed to the <codeph>impalad</codeph> daemons through the same
+        port. The <codeph>impalad</codeph> daemons listen on separate ports for JDBC and ODBC requests.
+      </p>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_disable_cached_reads.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_disable_cached_reads.xml b/docs/topics/impala_disable_cached_reads.xml
new file mode 100644
index 0000000..bce6091
--- /dev/null
+++ b/docs/topics/impala_disable_cached_reads.xml
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="disable_cached_reads" rev="1.4.0">
+
+  <title>DISABLE_CACHED_READS Query Option</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="HDFS"/>
+      <data name="Category" value="HDFS Caching"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">DISABLE_CACHED_READS query option</indexterm>
+      Prevents Impala from reading data files that are <q>pinned</q> in memory
+      through the HDFS caching feature. Primarily a debugging option for
+      cases where processing of HDFS cached data is concentrated on a single
+      host, leading to excessive CPU usage on that host.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_boolean"/>
+
+    <p conref="../shared/impala_common.xml#common/default_false"/>
+
+    <p conref="../shared/impala_common.xml#common/added_in_140"/>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_disable_codegen.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_disable_codegen.xml b/docs/topics/impala_disable_codegen.xml
new file mode 100644
index 0000000..ab306e8
--- /dev/null
+++ b/docs/topics/impala_disable_codegen.xml
@@ -0,0 +1,38 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="disable_codegen">
+
+  <title>DISABLE_CODEGEN Query Option</title>
+  <titlealts audience="PDF"><navtitle>DISABLE_CODEGEN</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Performance"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">DISABLE_CODEGEN query option</indexterm>
+      This is a debug option, intended for diagnosing and working around issues that cause crashes. If a query
+      fails with an <q>illegal instruction</q> or other hardware-specific message, try setting
+      <codeph>DISABLE_CODEGEN=true</codeph> and running the query again. If the query succeeds only when the
+      <codeph>DISABLE_CODEGEN</codeph> option is turned on, submit the problem to <keyword keyref="support_org"/> and include that
+      detail in the problem report. Do not otherwise run with this setting turned on, because it results in lower
+      overall performance.
+    </p>
+
+    <p>
+      Because the code generation phase adds a small amount of overhead for each query, you might turn on the
+      <codeph>DISABLE_CODEGEN</codeph> option to achieve maximum throughput when running many short-lived queries
+      against small tables.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_boolean"/>
+    <p conref="../shared/impala_common.xml#common/default_false_0"/>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_disable_outermost_topn.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_disable_outermost_topn.xml b/docs/topics/impala_disable_outermost_topn.xml
new file mode 100644
index 0000000..24fcd2a
--- /dev/null
+++ b/docs/topics/impala_disable_outermost_topn.xml
@@ -0,0 +1,29 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="disable_outermost_topn" rev="2.5.0">
+
+  <title>DISABLE_OUTERMOST_TOPN Query Option</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.5.0">
+      <indexterm audience="Cloudera">DISABLE_OUTERMOST_TOPN query option</indexterm>
+    </p>
+
+    <p>
+      <b>Type:</b>
+    </p>
+
+    <p>
+      <b>Default:</b>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_disable_row_runtime_filtering.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_disable_row_runtime_filtering.xml b/docs/topics/impala_disable_row_runtime_filtering.xml
new file mode 100644
index 0000000..a8c07cb
--- /dev/null
+++ b/docs/topics/impala_disable_row_runtime_filtering.xml
@@ -0,0 +1,65 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="disable_row_runtime_filtering" rev="2.5.0">
+
+  <title>DISABLE_ROW_RUNTIME_FILTERING Query Option (<keyword keyref="impala25"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>DISABLE_ROW_RUNTIME_FILTERING</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.5.0">
+      <indexterm audience="Cloudera">DISABLE_ROW_RUNTIME_FILTERING query option</indexterm>
+      The <codeph>DISABLE_ROW_RUNTIME_FILTERING</codeph> query option
+      reduces the scope of the runtime filtering feature. Queries still dynamically prune
+      partitions, but do not apply the filtering logic to individual rows within partitions.
+    </p>
+
+    <p>
+      Only applies to queries against Parquet tables. For other file formats, Impala
+      only prunes at the level of partitions, not individual rows.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_boolean"/>
+    <p conref="../shared/impala_common.xml#common/default_false"/>
+
+    <p conref="../shared/impala_common.xml#common/added_in_250"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      Impala automatically evaluates whether the per-row filters are being
+      effective at reducing the amount of intermediate data. Therefore,
+      this option is typically only needed for the rare case where Impala
+      cannot accurately determine how effective the per-row filtering is
+      for a query.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/runtime_filtering_option_caveat"/>
+
+    <p>
+      Because this setting only improves query performance in very specific
+      circumstances, depending on the query characteristics and data distribution,
+      only use it when you determine through benchmarking that it improves
+      performance of specific expensive queries.
+      Consider setting this query option immediately before the expensive query and
+      unsetting it immediately afterward.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_runtime_filtering.xml"/>,
+      <xref href="impala_runtime_filter_mode.xml#runtime_filter_mode"/>
+      <!-- , <xref href="impala_partitioning.xml#dynamic_partition_pruning"/> -->
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_disable_streaming_preaggregations.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_disable_streaming_preaggregations.xml b/docs/topics/impala_disable_streaming_preaggregations.xml
new file mode 100644
index 0000000..a3919af
--- /dev/null
+++ b/docs/topics/impala_disable_streaming_preaggregations.xml
@@ -0,0 +1,45 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="disable_streaming_preaggregations" rev="2.5.0 IMPALA-1305">
+
+  <title>DISABLE_STREAMING_PREAGGREGATIONS Query Option (<keyword keyref="impala25"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>DISABLE_STREAMING_PREAGGREGATIONS</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Aggregate Functions"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.5.0 IMPALA-1305">
+      <indexterm audience="Cloudera">DISABLE_STREAMING_PREAGGREGATIONS query option</indexterm>
+      Turns off the <q>streaming preaggregation</q> optimization that is available in <keyword keyref="impala25_full"/>
+      and higher. This optimization reduces unnecessary work performed by queries that perform aggregation
+      operations on columns with few or no duplicate values, for example <codeph>DISTINCT <varname>id_column</varname></codeph>
+      or <codeph>GROUP BY <varname>unique_column</varname></codeph>. If the optimization causes regressions in
+      existing queries that use aggregation functions, you can turn it off as needed by setting this query option.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_boolean"/>
+    <p conref="../shared/impala_common.xml#common/default_false_0"/>
+
+    <note conref="../shared/impala_common.xml#common/one_but_not_true"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+    <p>
+      Typically, queries that would require enabling this option involve very large numbers of
+      aggregated values, such as a billion or more distinct keys being processed on each
+      worker node.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/added_in_250"/>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_disable_unsafe_spills.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_disable_unsafe_spills.xml b/docs/topics/impala_disable_unsafe_spills.xml
new file mode 100644
index 0000000..14b687c
--- /dev/null
+++ b/docs/topics/impala_disable_unsafe_spills.xml
@@ -0,0 +1,53 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="2.0.0" id="disable_unsafe_spills">
+
+  <title>DISABLE_UNSAFE_SPILLS Query Option (<keyword keyref="impala20"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>DISABLE_UNSAFE_SPILLS</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Scalability"/>
+      <data name="Category" value="Memory"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.0.0">
+      <indexterm audience="Cloudera">DISABLE_UNSAFE_SPILLS query option</indexterm>
+      Enable this option if you prefer to have queries fail when they exceed the Impala memory limit, rather than
+      write temporary data to disk.
+    </p>
+
+    <p>
+      Queries that <q>spill</q> to disk typically complete successfully, when in earlier Impala releases they would have failed.
+      However, queries with exorbitant memory requirements due to missing statistics or inefficient join clauses could
+      become so slow as a result that you would rather have them cancelled automatically and reduce the memory
+      usage through standard Impala tuning techniques.
+    </p>
+
+    <p>
+      This option prevents only <q>unsafe</q> spill operations, meaning that one or more tables are missing
+      statistics or the query does not include a hint to set the most efficient mechanism for a join or
+      <codeph>INSERT ... SELECT</codeph> into a partitioned table. These are the tables most likely to result in
+      suboptimal execution plans that could cause unnecessary spilling. Therefore, leaving this option enabled is a
+      good way to find tables on which to run the <codeph>COMPUTE STATS</codeph> statement.
+    </p>
+
+    <p>
+      See <xref href="impala_scalability.xml#spill_to_disk"/> for information about the <q>spill to disk</q>
+      feature for queries processing large result sets with joins, <codeph>ORDER BY</codeph>, <codeph>GROUP
+      BY</codeph>, <codeph>DISTINCT</codeph>, aggregation functions, or analytic functions.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_boolean"/>
+    <p conref="../shared/impala_common.xml#common/default_false_0"/>
+
+    <p conref="../shared/impala_common.xml#common/added_in_20"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_disk_space.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_disk_space.xml b/docs/topics/impala_disk_space.xml
new file mode 100644
index 0000000..b6daaeb
--- /dev/null
+++ b/docs/topics/impala_disk_space.xml
@@ -0,0 +1,129 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="disk_space">
+
+  <title>Managing Disk Space for Impala Data</title>
+  <titlealts audience="PDF"><navtitle>Managing Disk Space</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Disk Storage"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Compression"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Although Impala typically works with many large files in an HDFS storage system with plenty of capacity,
+      there are times when you might perform some file cleanup to reclaim space, or advise developers on techniques
+      to minimize space consumption and file duplication.
+    </p>
+
+    <ul>
+      <li>
+        <p>
+          Use compact binary file formats where practical. Numeric and time-based data in particular can be stored
+          in more compact form in binary data files. Depending on the file format, various compression and encoding
+          features can reduce file size even further. You can specify the <codeph>STORED AS</codeph> clause as part
+          of the <codeph>CREATE TABLE</codeph> statement, or <codeph>ALTER TABLE</codeph> with the <codeph>SET
+          FILEFORMAT</codeph> clause for an existing table or partition within a partitioned table. See
+          <xref href="impala_file_formats.xml#file_formats"/> for details about file formats, especially
+          <xref href="impala_parquet.xml#parquet"/>. See <xref href="impala_create_table.xml#create_table"/> and
+          <xref href="impala_alter_table.xml#alter_table"/> for syntax details.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          You manage underlying data files differently depending on whether the corresponding Impala table is
+          defined as an <xref href="impala_tables.xml#internal_tables">internal</xref> or
+          <xref href="impala_tables.xml#external_tables">external</xref> table:
+        </p>
+        <ul>
+          <li>
+            Use the <codeph>DESCRIBE FORMATTED</codeph> statement to check if a particular table is internal
+            (managed by Impala) or external, and to see the physical location of the data files in HDFS. See
+            <xref href="impala_describe.xml#describe"/> for details.
+          </li>
+
+          <li>
+            For Impala-managed (<q>internal</q>) tables, use <codeph>DROP TABLE</codeph> statements to remove
+            data files. See <xref href="impala_drop_table.xml#drop_table"/> for details.
+          </li>
+
+          <li>
+            For tables not managed by Impala (<q>external</q> tables), use appropriate HDFS-related commands such
+            as <codeph>hadoop fs</codeph>, <codeph>hdfs dfs</codeph>, or <codeph>distcp</codeph>, to create, move,
+            copy, or delete files within HDFS directories that are accessible by the <codeph>impala</codeph> user.
+            Issue a <codeph>REFRESH <varname>table_name</varname></codeph> statement after adding or removing any
+            files from the data directory of an external table. See <xref href="impala_refresh.xml#refresh"/> for
+            details.
+          </li>
+
+          <li>
+            Use external tables to reference HDFS data files in their original location. With this technique, you
+            avoid copying the files, and you can map more than one Impala table to the same set of data files. When
+            you drop the Impala table, the data files are left undisturbed. See
+            <xref href="impala_tables.xml#external_tables"/> for details.
+          </li>
+
+          <li>
+            Use the <codeph>LOAD DATA</codeph> statement to move HDFS files into the data directory for an Impala
+            table from inside Impala, without the need to specify the HDFS path of the destination directory. This
+            technique works for both internal and external tables. See
+            <xref href="impala_load_data.xml#load_data"/> for details.
+          </li>
+        </ul>
+      </li>
+
+      <li>
+        <p>
+          Make sure that the HDFS trashcan is configured correctly. When you remove files from HDFS, the space
+          might not be reclaimed for use by other files until sometime later, when the trashcan is emptied. See
+          <xref href="impala_drop_table.xml#drop_table"/> and the FAQ entry
+          <xref href="impala_faq.xml#faq_sql/faq_drop_table_space"/> for details. See
+          <xref href="impala_prereqs.xml#prereqs_account"/> for permissions needed for the HDFS trashcan to operate
+          correctly.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          Drop all tables in a database before dropping the database itself. See
+          <xref href="impala_drop_database.xml#drop_database"/> for details.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          Clean up temporary files after failed <codeph>INSERT</codeph> statements. If an <codeph>INSERT</codeph>
+          statement encounters an error, and you see a directory named <filepath>.impala_insert_staging</filepath>
+          or <filepath>_impala_insert_staging</filepath> left behind in the data directory for the table, it might
+          contain temporary data files taking up space in HDFS. You might be able to salvage these data files, for
+          example if they are complete but could not be moved into place due to a permission error. Or, you might
+          delete those files through commands such as <codeph>hadoop fs</codeph> or <codeph>hdfs dfs</codeph>, to
+          reclaim space before re-trying the <codeph>INSERT</codeph>. Issue <codeph>DESCRIBE FORMATTED
+          <varname>table_name</varname></codeph> to see the HDFS path where you can check for temporary files.
+        </p>
+      </li>
+
+      <li rev="1.4.0">
+        <p rev="obwl" conref="../shared/impala_common.xml#common/order_by_scratch_dir"/>
+      </li>
+
+      <li rev="2.2.0">
+        <p>
+          If you use the Amazon Simple Storage Service (S3) as a place to offload
+          data to reduce the volume of local storage, Impala 2.2.0 and higher
+          can query the data directly from S3.
+          See <xref href="impala_s3.xml#s3"/> for details.
+        </p>
+      </li>
+    </ul>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_distinct.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_distinct.xml b/docs/topics/impala_distinct.xml
new file mode 100644
index 0000000..8661831
--- /dev/null
+++ b/docs/topics/impala_distinct.xml
@@ -0,0 +1,61 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="distinct">
+
+  <title>DISTINCT Operator</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Aggregate Functions"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">DISTINCT operator</indexterm>
+      The <codeph>DISTINCT</codeph> operator in a <codeph>SELECT</codeph> statement filters the result set to
+      remove duplicates:
+    </p>
+
+<codeblock>-- Returns the unique values from one column.
+-- NULL is included in the set of values if any rows have a NULL in this column.
+select distinct c_birth_country from customer;
+-- Returns the unique combinations of values from multiple columns.
+select distinct c_salutation, c_last_name from customer;</codeblock>
+
+    <p>
+      You can use <codeph>DISTINCT</codeph> in combination with an aggregation function, typically
+      <codeph>COUNT()</codeph>, to find how many different values a column contains:
+    </p>
+
+<codeblock>-- Counts the unique values from one column.
+-- NULL is not included as a distinct value in the count.
+select count(distinct c_birth_country) from customer;
+-- Counts the unique combinations of values from multiple columns.
+select count(distinct c_salutation, c_last_name) from customer;</codeblock>
+
+    <p>
+      One construct that Impala SQL does <i>not</i> support is using <codeph>DISTINCT</codeph> in more than one
+      aggregation function in the same query. For example, you could not have a single query with both
+      <codeph>COUNT(DISTINCT c_first_name)</codeph> and <codeph>COUNT(DISTINCT c_last_name)</codeph> in the
+      <codeph>SELECT</codeph> list.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/zero_length_strings"/>
+
+    <note conref="../shared/impala_common.xml#common/multiple_count_distinct"/>
+
+    <note>
+      <p>
+        In contrast with some database systems that always return <codeph>DISTINCT</codeph> values in sorted order,
+        Impala does not do any ordering of <codeph>DISTINCT</codeph> values. Always include an <codeph>ORDER
+        BY</codeph> clause if you need the values in alphabetical or numeric sorted order.
+      </p>
+    </note>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_dml.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_dml.xml b/docs/topics/impala_dml.xml
new file mode 100644
index 0000000..b6c6399
--- /dev/null
+++ b/docs/topics/impala_dml.xml
@@ -0,0 +1,91 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="dml">
+
+  <title>DML Statements</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="DML"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="ETL"/>
+      <data name="Category" value="Ingest"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      DML refers to <q>Data Manipulation Language</q>, a subset of SQL statements that modify the data stored in
+      tables. Because Impala focuses on query performance and leverages the append-only nature of HDFS storage,
+      currently Impala only supports a small set of DML statements:
+    </p>
+
+    <ul>
+      <li>
+        <xref keyref="delete"/>. Works for Kudu tables only.
+      </li>
+
+      <li>
+        <xref keyref="insert"/>.
+      </li>
+
+      <li>
+        <xref keyref="load_data"/>. Does not apply for HBase or Kudu tables.
+      </li>
+
+      <li>
+        <xref keyref="update"/>. Works for Kudu tables only.
+      </li>
+
+      <li>
+        <xref keyref="upsert"/>. Works for Kudu tables only.
+      </li>
+    </ul>
+
+    <p>
+      <codeph>INSERT</codeph> in Impala is primarily optimized for inserting large volumes of data in a single
+      statement, to make effective use of the multi-megabyte HDFS blocks. This is the way in Impala to create new
+      data files. If you intend to insert one or a few rows at a time, such as using the <codeph>INSERT ...
+      VALUES</codeph> syntax, that technique is much more efficient for Impala tables stored in HBase. See
+      <xref href="impala_hbase.xml#impala_hbase"/> for details.
+    </p>
+
+    <p>
+      <codeph>LOAD DATA</codeph> moves existing data files into the directory for an Impala table, making them
+      immediately available for Impala queries. This is one way in Impala to work with data files produced by other
+      Hadoop components. (<codeph>CREATE EXTERNAL TABLE</codeph> is the other alternative; with external tables,
+      you can query existing data files, while the files remain in their original location.)
+    </p>
+
+    <p>
+      In <keyword keyref="impala28_full"/> and higher, Impala does support the <codeph>UPDATE</codeph>, <codeph>DELETE</codeph>,
+      and <codeph>UPSERT</codeph> statements for Kudu tables.
+      For HDFS or S3 tables, to simulate the effects of an <codeph>UPDATE</codeph> or <codeph>DELETE</codeph> statement
+      in other database systems, typically you use <codeph>INSERT</codeph> or <codeph>CREATE TABLE AS SELECT</codeph> to copy data
+      from one table to another, filtering out or changing the appropriate rows during the copy operation.
+    </p>
+
+    <p>
+      You can also achieve a result similar to <codeph>UPDATE</codeph> by using Impala tables stored in HBase.
+      When you insert a row into an HBase table, and the table
+      already contains a row with the same value for the key column, the older row is hidden, effectively the same
+      as a single-row <codeph>UPDATE</codeph>.
+    </p>
+
+    <p rev="2.6.0">
+      Impala can perform DML operations for tables or partitions stored in the Amazon S3 filesystem
+      with <keyword keyref="impala26_full"/> and higher. See <xref href="impala_s3.xml#s3"/> for details.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      The other major classifications of SQL statements are data definition language (see
+      <xref href="impala_ddl.xml#ddl"/>) and queries (see <xref href="impala_select.xml#select"/>).
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_double.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_double.xml b/docs/topics/impala_double.xml
new file mode 100644
index 0000000..c69eae2
--- /dev/null
+++ b/docs/topics/impala_double.xml
@@ -0,0 +1,100 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="double">
+
+  <title>DOUBLE Data Type</title>
+  <titlealts audience="PDF"><navtitle>DOUBLE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      A double precision floating-point data type used in <codeph>CREATE TABLE</codeph> and <codeph>ALTER
+      TABLE</codeph> statements.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p>
+      In the column definition of a <codeph>CREATE TABLE</codeph> statement:
+    </p>
+
+<codeblock><varname>column_name</varname> DOUBLE</codeblock>
+
+    <p>
+      <b>Range:</b> 4.94065645841246544e-324d .. 1.79769313486231570e+308, positive or negative
+    </p>
+
+    <p>
+      <b>Precision:</b> 15 to 17 significant digits, depending on usage. The number of significant digits does
+      not depend on the position of the decimal point.
+    </p>
+
+    <p>
+      <b>Representation:</b> The values are stored in 8 bytes, using
+      <xref href="https://en.wikipedia.org/wiki/Double-precision_floating-point_format" scope="external" format="html">IEEE 754 Double Precision Binary Floating Point</xref> format.
+    </p>
+
+    <p>
+      <b>Conversions:</b> Impala does not automatically convert <codeph>DOUBLE</codeph> to any other type. You can
+      use <codeph>CAST()</codeph> to convert <codeph>DOUBLE</codeph> values to <codeph>FLOAT</codeph>,
+      <codeph>TINYINT</codeph>, <codeph>SMALLINT</codeph>, <codeph>INT</codeph>, <codeph>BIGINT</codeph>,
+      <codeph>STRING</codeph>, <codeph>TIMESTAMP</codeph>, or <codeph>BOOLEAN</codeph>. You can use exponential
+      notation in <codeph>DOUBLE</codeph> literals or when casting from <codeph>STRING</codeph>, for example
+      <codeph>1.0e6</codeph> to represent one million.
+      <ph conref="../shared/impala_common.xml#common/cast_int_to_timestamp"/>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      The data type <codeph>REAL</codeph> is an alias for <codeph>DOUBLE</codeph>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>CREATE TABLE t1 (x DOUBLE);
+SELECT CAST(1000.5 AS DOUBLE);
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/partitioning_imprecise"/>
+
+    <p conref="../shared/impala_common.xml#common/hbase_ok"/>
+
+    <p conref="../shared/impala_common.xml#common/parquet_ok"/>
+
+    <p conref="../shared/impala_common.xml#common/text_bulky"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/compatibility_blurb"/> -->
+
+    <p conref="../shared/impala_common.xml#common/internals_8_bytes"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/added_in_20"/> -->
+
+    <p conref="../shared/impala_common.xml#common/column_stats_constant"/>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+<!-- This conref appears under SUM(), AVG(), FLOAT, and DOUBLE topics. -->
+
+    <p conref="../shared/impala_common.xml#common/sum_double"/>
+
+    <p conref="../shared/impala_common.xml#common/float_double_decimal_caveat"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_literals.xml#numeric_literals"/>, <xref href="impala_math_functions.xml#math_functions"/>,
+      <xref href="impala_float.xml#float"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_drop_data_source.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_drop_data_source.xml b/docs/topics/impala_drop_data_source.xml
new file mode 100644
index 0000000..be6167c
--- /dev/null
+++ b/docs/topics/impala_drop_data_source.xml
@@ -0,0 +1,35 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept audience="Cloudera" rev="1.4.0" id="drop_data_source">
+
+  <title>DROP DATA SOURCE Statement</title>
+  <titlealts audience="PDF"><navtitle>DROP DATA SOURCE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">DROP DATA SOURCE statement</indexterm>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_drop_database.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_drop_database.xml b/docs/topics/impala_drop_database.xml
new file mode 100644
index 0000000..fb36dca
--- /dev/null
+++ b/docs/topics/impala_drop_database.xml
@@ -0,0 +1,130 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="drop_database">
+
+  <title>DROP DATABASE Statement</title>
+  <titlealts audience="PDF"><navtitle>DROP DATABASE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Databases"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="Schemas"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">DROP DATABASE statement</indexterm>
+      Removes a database from the system. The physical operations involve removing the metadata for the database
+      from the metastore, and deleting the corresponding <codeph>*.db</codeph> directory from HDFS.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>DROP (DATABASE|SCHEMA) [IF EXISTS] <varname>database_name</varname> <ph rev="2.3.0">[RESTRICT | CASCADE]</ph>;</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      By default, the database must be empty before it can be dropped, to avoid losing any data.
+    </p>
+
+    <p rev="2.3.0">
+      In <keyword keyref="impala23_full"/> and higher, you can include the <codeph>CASCADE</codeph>
+      clause to make Impala drop all tables and other objects in the database before dropping the database itself.
+      The <codeph>RESTRICT</codeph> clause enforces the original requirement that the database be empty
+      before being dropped. Because the <codeph>RESTRICT</codeph> behavior is still the default, this
+      clause is optional.
+    </p>
+
+    <p rev="2.3.0">
+      The automatic dropping resulting from the <codeph>CASCADE</codeph> clause follows the same rules as the
+      corresponding <codeph>DROP TABLE</codeph>, <codeph>DROP VIEW</codeph>, and <codeph>DROP FUNCTION</codeph> statements.
+      In particular, the HDFS directories and data files for any external tables are left behind when the
+      tables are removed.
+    </p>
+
+    <p>
+      When you do not use the <codeph>CASCADE</codeph> clause, drop or move all the objects inside the database manually
+      before dropping the database itself:
+    </p>
+
+    <ul>
+      <li>
+        <p>
+          Use the <codeph>SHOW TABLES</codeph> statement to locate all tables and views in the database,
+          and issue <codeph>DROP TABLE</codeph> and <codeph>DROP VIEW</codeph> statements to remove them all.
+        </p>
+      </li>
+      <li>
+        <p>
+          Use the <codeph>SHOW FUNCTIONS</codeph> and <codeph>SHOW AGGREGATE FUNCTIONS</codeph> statements
+          to locate all user-defined functions in the database, and issue <codeph>DROP FUNCTION</codeph>
+          and <codeph>DROP AGGREGATE FUNCTION</codeph> statements to remove them all.
+        </p>
+      </li>
+      <li>
+        <p>
+          To keep tables or views contained by a database while removing the database itself, use
+          <codeph>ALTER TABLE</codeph> and <codeph>ALTER VIEW</codeph> to move the relevant
+          objects to a different database before dropping the original database.
+        </p>
+      </li>
+    </ul>
+
+    <p>
+      You cannot drop the current database, that is, the database your session connected to
+      either through the <codeph>USE</codeph> statement or the <codeph>-d</codeph> option of <cmdname>impala-shell</cmdname>.
+      Issue a <codeph>USE</codeph> statement to switch to a different database first.
+      Because the <codeph>default</codeph> database is always available, issuing
+      <codeph>USE default</codeph> is a convenient way to leave the current database
+      before dropping it.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/hive_blurb"/>
+
+    <p>
+      When you drop a database in Impala, the database can no longer be used by Hive.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<!-- Better to conref the same examples in both places. -->
+
+    <p>
+      See <xref href="impala_create_database.xml#create_database"/> for examples covering <codeph>CREATE
+      DATABASE</codeph>, <codeph>USE</codeph>, and <codeph>DROP DATABASE</codeph>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/s3_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/s3_ddl"/>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+    <p rev="CDH-19187">
+      The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+      typically the <codeph>impala</codeph> user, must have write
+      permission for the directory associated with the database.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <codeblock conref="../shared/impala_common.xml#common/create_drop_db_example"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_databases.xml#databases"/>, <xref href="impala_create_database.xml#create_database"/>,
+      <xref href="impala_use.xml#use"/>, <xref href="impala_show.xml#show_databases"/>, <xref href="impala_drop_table.xml#drop_table"/>
+    </p>
+  </conbody>
+</concept>


[38/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_config_options.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_config_options.xml b/docs/topics/impala_config_options.xml
new file mode 100644
index 0000000..a08aafa
--- /dev/null
+++ b/docs/topics/impala_config_options.xml
@@ -0,0 +1,593 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="config_options">
+
+  <title>Modifying Impala Startup Options</title>
+
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Configuring"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">defaults file</indexterm>
+
+      <indexterm audience="Cloudera">configuration file</indexterm>
+
+      <indexterm audience="Cloudera">options</indexterm>
+
+      <indexterm audience="Cloudera">IMPALA_STATE_STORE_PORT</indexterm>
+
+      <indexterm audience="Cloudera">IMPALA_BACKEND_PORT</indexterm>
+
+      <indexterm audience="Cloudera">IMPALA_LOG_DIR</indexterm>
+
+      <indexterm audience="Cloudera">IMPALA_STATE_STORE_ARGS</indexterm>
+
+      <indexterm audience="Cloudera">IMPALA_SERVER_ARGS</indexterm>
+
+      <indexterm audience="Cloudera">ENABLE_CORE_DUMPS</indexterm>
+
+      <indexterm audience="Cloudera">core dumps</indexterm>
+
+      <indexterm audience="Cloudera">restarting services</indexterm>
+
+      <indexterm audience="Cloudera">services</indexterm>
+      The configuration options for the Impala-related daemons let you choose which hosts and
+      ports to use for the services that run on a single host, specify directories for logging,
+      control resource usage and security, and specify other aspects of the Impala software.
+    </p>
+
+    <p outputclass="toc inpage"/>
+
+  </conbody>
+
+  <concept id="config_options_cm">
+
+    <title>Configuring Impala Startup Options through Cloudera Manager</title>
+
+    <conbody>
+
+      <p>
+        If you manage your cluster through Cloudera Manager, configure the settings for all the
+        Impala-related daemons by navigating to this page:
+        <menucascade><uicontrol>Clusters</uicontrol><uicontrol>Impala</uicontrol><uicontrol>Configuration</uicontrol><uicontrol>View
+        and Edit</uicontrol></menucascade>. See the Cloudera Manager documentation for
+        <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_impala_service.html" scope="external" format="html">instructions
+        about how to configure Impala through Cloudera Manager</xref>.
+      </p>
+
+      <p>
+        If the Cloudera Manager interface does not yet have a form field for a newly added
+        option, or if you need to use special options for debugging and troubleshooting, the
+        <uicontrol>Advanced</uicontrol> option page for each daemon includes one or more fields
+        where you can enter option names directly.
+        <ph conref="../shared/impala_common.xml#common/safety_valve"/> There is also a free-form
+        field for query options, on the top-level <uicontrol>Impala Daemon</uicontrol> options
+        page.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="config_options_noncm">
+
+    <title>Configuring Impala Startup Options through the Command Line</title>
+
+    <conbody>
+
+      <p>
+        When you run Impala in a non-Cloudera Manager environment, the Impala server,
+        statestore, and catalog services start up using values provided in a defaults file,
+        <filepath>/etc/default/impala</filepath>.
+      </p>
+
+      <p>
+        This file includes information about many resources used by Impala. Most of the defaults
+        included in this file should be effective in most cases. For example, typically you
+        would not change the definition of the <codeph>CLASSPATH</codeph> variable, but you
+        would always set the address used by the statestore server. Some of the content you
+        might modify includes:
+      </p>
+
+<!-- Note: Update the following example for each release with the associated lines from /etc/default/impala
+           from a non-CM-managed system. -->
+
+<codeblock rev="ver">IMPALA_STATE_STORE_HOST=127.0.0.1
+IMPALA_STATE_STORE_PORT=24000
+IMPALA_BACKEND_PORT=22000
+IMPALA_LOG_DIR=/var/log/impala
+IMPALA_CATALOG_SERVICE_HOST=...
+IMPALA_STATE_STORE_HOST=...
+
+export IMPALA_STATE_STORE_ARGS=${IMPALA_STATE_STORE_ARGS:- \
+    -log_dir=${IMPALA_LOG_DIR} -state_store_port=${IMPALA_STATE_STORE_PORT}}
+IMPALA_SERVER_ARGS=" \
+-log_dir=${IMPALA_LOG_DIR} \
+-catalog_service_host=${IMPALA_CATALOG_SERVICE_HOST} \
+-state_store_port=${IMPALA_STATE_STORE_PORT} \
+-use_statestore \
+-state_store_host=${IMPALA_STATE_STORE_HOST} \
+-be_port=${IMPALA_BACKEND_PORT}"
+export ENABLE_CORE_DUMPS=${ENABLE_COREDUMPS:-false}</codeblock>
+
+      <p>
+        To use alternate values, edit the defaults file, then restart all the Impala-related
+        services so that the changes take effect. Restart the Impala server using the following
+        commands:
+      </p>
+
+<codeblock>$ sudo service impala-server restart
+Stopping Impala Server:                                    [  OK  ]
+Starting Impala Server:                                    [  OK  ]</codeblock>
+
+      <p>
+        Restart the Impala statestore using the following commands:
+      </p>
+
+<codeblock>$ sudo service impala-state-store restart
+Stopping Impala State Store Server:                        [  OK  ]
+Starting Impala State Store Server:                        [  OK  ]</codeblock>
+
+      <p>
+        Restart the Impala catalog service using the following commands:
+      </p>
+
+<codeblock>$ sudo service impala-catalog restart
+Stopping Impala Catalog Server:                            [  OK  ]
+Starting Impala Catalog Server:                            [  OK  ]</codeblock>
+
+      <p>
+        Some common settings to change include:
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            Statestore address. Where practical, put the statestore on a separate host not
+            running the <cmdname>impalad</cmdname> daemon. In that recommended configuration,
+            the <cmdname>impalad</cmdname> daemon cannot refer to the statestore server using
+            the loopback address. If the statestore is hosted on a machine with an IP address of
+            192.168.0.27, change:
+          </p>
+<codeblock>IMPALA_STATE_STORE_HOST=127.0.0.1</codeblock>
+          <p>
+            to:
+          </p>
+<codeblock>IMPALA_STATE_STORE_HOST=192.168.0.27</codeblock>
+        </li>
+
+        <li rev="1.2">
+          <p>
+            Catalog server address (including both the hostname and the port number). Update the
+            value of the <codeph>IMPALA_CATALOG_SERVICE_HOST</codeph> variable. Cloudera
+            recommends the catalog server be on the same host as the statestore. In that
+            recommended configuration, the <cmdname>impalad</cmdname> daemon cannot refer to the
+            catalog server using the loopback address. If the catalog service is hosted on a
+            machine with an IP address of 192.168.0.27, add the following line:
+          </p>
+<codeblock>IMPALA_CATALOG_SERVICE_HOST=192.168.0.27:26000</codeblock>
+          <p>
+            The <filepath>/etc/default/impala</filepath> defaults file currently does not define
+            an <codeph>IMPALA_CATALOG_ARGS</codeph> environment variable, but if you add one it
+            will be recognized by the service startup/shutdown script. Add a definition for this
+            variable to <filepath>/etc/default/impala</filepath> and add the option
+            <codeph>-catalog_service_host=<varname>hostname</varname></codeph>. If the port is
+            different than the default 26000, also add the option
+            <codeph>-catalog_service_port=<varname>port</varname></codeph>.
+          </p>
+        </li>
+
+        <li id="mem_limit">
+          <p>
+            Memory limits. You can limit the amount of memory available to Impala. For example,
+            to allow Impala to use no more than 70% of system memory, change:
+          </p>
+<!-- Note: also needs to be updated for each release to reflect latest /etc/default/impala. -->
+<codeblock>export IMPALA_SERVER_ARGS=${IMPALA_SERVER_ARGS:- \
+    -log_dir=${IMPALA_LOG_DIR} \
+    -state_store_port=${IMPALA_STATE_STORE_PORT} \
+    -use_statestore -state_store_host=${IMPALA_STATE_STORE_HOST} \
+    -be_port=${IMPALA_BACKEND_PORT}}</codeblock>
+          <p>
+            to:
+          </p>
+<codeblock>export IMPALA_SERVER_ARGS=${IMPALA_SERVER_ARGS:- \
+    -log_dir=${IMPALA_LOG_DIR} -state_store_port=${IMPALA_STATE_STORE_PORT} \
+    -use_statestore -state_store_host=${IMPALA_STATE_STORE_HOST} \
+    -be_port=${IMPALA_BACKEND_PORT} -mem_limit=70%}</codeblock>
+          <p>
+            You can specify the memory limit using absolute notation such as
+            <codeph>500m</codeph> or <codeph>2G</codeph>, or as a percentage of physical memory
+            such as <codeph>60%</codeph>.
+          </p>
+
+          <note>
+            Queries that exceed the specified memory limit are aborted. Percentage limits are
+            based on the physical memory of the machine and do not consider cgroups.
+          </note>
+        </li>
+
+        <li>
+          <p>
+            Core dump enablement. To enable core dumps on systems not managed by Cloudera
+            Manager, change:
+          </p>
+<codeblock>export ENABLE_CORE_DUMPS=${ENABLE_COREDUMPS:-false}</codeblock>
+          <p>
+            to:
+          </p>
+<codeblock>export ENABLE_CORE_DUMPS=${ENABLE_COREDUMPS:-true}</codeblock>
+          <p>
+            On systems managed by Cloudera Manager, enable the <uicontrol>Enable Core
+            Dump</uicontrol> setting for the Impala service.
+          </p>
+
+          <note conref="../shared/impala_common.xml#common/core_dump_considerations"/>
+        </li>
+
+        <li>
+          <p>
+            Authorization using the open source Sentry plugin. Specify the
+            <codeph>-server_name</codeph> and <codeph>-authorization_policy_file</codeph>
+            options as part of the <codeph>IMPALA_SERVER_ARGS</codeph> and
+            <codeph>IMPALA_STATE_STORE_ARGS</codeph> settings to enable the core Impala support
+            for authentication. See <xref href="impala_authorization.xml#secure_startup"/> for
+            details.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Auditing for successful or blocked Impala queries, another aspect of security.
+            Specify the <codeph>-audit_event_log_dir=<varname>directory_path</varname></codeph>
+            option and optionally the
+            <codeph>-max_audit_event_log_file_size=<varname>number_of_queries</varname></codeph>
+            and <codeph>-abort_on_failed_audit_event</codeph> options as part of the
+            <codeph>IMPALA_SERVER_ARGS</codeph> settings, for each Impala node, to enable and
+            customize auditing. See <xref href="impala_auditing.xml#auditing"/> for details.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Password protection for the Impala web UI, which listens on port 25000 by default.
+            This feature involves adding some or all of the
+            <codeph>--webserver_password_file</codeph>,
+            <codeph>--webserver_authentication_domain</codeph>, and
+            <codeph>--webserver_certificate_file</codeph> options to the
+            <codeph>IMPALA_SERVER_ARGS</codeph> and <codeph>IMPALA_STATE_STORE_ARGS</codeph>
+            settings. See <xref href="impala_security_guidelines.xml#security_guidelines"/> for
+            details.
+          </p>
+        </li>
+
+        <li id="default_query_options">
+          <p rev="DOCS-677">
+            Another setting you might add to <codeph>IMPALA_SERVER_ARGS</codeph> is a
+            comma-separated list of query options and values:
+<codeblock>-default_query_options='<varname>option</varname>=<varname>value</varname>,<varname>option</varname>=<varname>value</varname>,...'
+</codeblock>
+            These options control the behavior of queries performed by this
+            <cmdname>impalad</cmdname> instance. The option values you specify here override the
+            default values for <xref href="impala_query_options.xml#query_options">Impala query
+            options</xref>, as shown by the <codeph>SET</codeph> statement in
+            <cmdname>impala-shell</cmdname>.
+          </p>
+        </li>
+
+<!-- Removing this reference now that the options are de-emphasized / desupported in CDH 5.5 / Impala 2.3 and up.
+        <li rev="1.2">
+          <p>
+          Options for resource management, in conjunction with the YARN component. These options include
+          <codeph>-enable_rm</codeph> and <codeph>-cgroup_hierarchy_path</codeph>.
+          <ph rev="1.4.0">Additional options to help fine-tune the resource estimates are
+          <codeph>-\u2014rm_always_use_defaults</codeph>,
+          <codeph>-\u2014rm_default_memory=<varname>size</varname></codeph>, and
+          <codeph>-\u2014rm_default_cpu_cores</codeph>.</ph> For details about these options, see
+          <xref href="impala_resource_management.xml#rm_options"/>. See
+          <xref href="impala_resource_management.xml#resource_management"/> for information about resource
+          management in general.
+          </p>
+        </li>
+-->
+
+        <li>
+          <p>
+            During troubleshooting, <keyword keyref="support_org"/> might direct you to change other values,
+            particularly for <codeph>IMPALA_SERVER_ARGS</codeph>, to work around issues or
+            gather debugging information.
+          </p>
+        </li>
+      </ul>
+
+<!-- Removing this reference now that the options are de-emphasized / desupported in CDH 5.5 / Impala 2.3 and up.
+      <p conref="impala_resource_management.xml#rm_options/resource_management_impalad_options"/>
+-->
+
+      <note>
+        <p>
+          These startup options for the <cmdname>impalad</cmdname> daemon are different from the
+          command-line options for the <cmdname>impala-shell</cmdname> command. For the
+          <cmdname>impala-shell</cmdname> options, see
+          <xref href="impala_shell_options.xml#shell_options"/>.
+        </p>
+      </note>
+
+      <p audience="Cloudera" outputclass="toc inpage"/>
+
+    </conbody>
+
+    <concept audience="Cloudera" id="config_options_impalad_details">
+
+      <title>Configuration Options for impalad Daemon</title>
+
+      <conbody>
+
+        <p>
+          Some common settings to change include:
+        </p>
+
+        <ul>
+          <li>
+            <p>
+              Statestore address. Where practical, put the statestore on a separate host not
+              running the <cmdname>impalad</cmdname> daemon. In that recommended configuration,
+              the <cmdname>impalad</cmdname> daemon cannot refer to the statestore server using
+              the loopback address. If the statestore is hosted on a machine with an IP address
+              of 192.168.0.27, change:
+            </p>
+<codeblock>IMPALA_STATE_STORE_HOST=127.0.0.1</codeblock>
+            <p>
+              to:
+            </p>
+<codeblock>IMPALA_STATE_STORE_HOST=192.168.0.27</codeblock>
+          </li>
+
+          <li rev="1.2">
+            <p>
+              Catalog server address. Update the <codeph>IMPALA_CATALOG_SERVICE_HOST</codeph>
+              variable, including both the hostname and the port number in the value. Cloudera
+              recommends the catalog server be on the same host as the statestore. In that
+              recommended configuration, the <cmdname>impalad</cmdname> daemon cannot refer to
+              the catalog server using the loopback address. If the catalog service is hosted on
+              a machine with an IP address of 192.168.0.27, add the following line:
+            </p>
+<codeblock>IMPALA_CATALOG_SERVICE_HOST=192.168.0.27:26000</codeblock>
+            <p>
+              The <filepath>/etc/default/impala</filepath> defaults file currently does not
+              define an <codeph>IMPALA_CATALOG_ARGS</codeph> environment variable, but if you
+              add one it will be recognized by the service startup/shutdown script. Add a
+              definition for this variable to <filepath>/etc/default/impala</filepath> and add
+              the option <codeph>-catalog_service_host=<varname>hostname</varname></codeph>. If
+              the port is different than the default 26000, also add the option
+              <codeph>-catalog_service_port=<varname>port</varname></codeph>.
+            </p>
+          </li>
+
+          <li id="mem_limit">
+            Memory limits. You can limit the amount of memory available to Impala. For example,
+            to allow Impala to use no more than 70% of system memory, change:
+<!-- Note: also needs to be updated for each release to reflect latest /etc/default/impala. -->
+<codeblock>export IMPALA_SERVER_ARGS=${IMPALA_SERVER_ARGS:- \
+    -log_dir=${IMPALA_LOG_DIR} \
+    -state_store_port=${IMPALA_STATE_STORE_PORT} \
+    -use_statestore -state_store_host=${IMPALA_STATE_STORE_HOST} \
+    -be_port=${IMPALA_BACKEND_PORT}}</codeblock>
+            <p>
+              to:
+            </p>
+<codeblock>export IMPALA_SERVER_ARGS=${IMPALA_SERVER_ARGS:- \
+    -log_dir=${IMPALA_LOG_DIR} -state_store_port=${IMPALA_STATE_STORE_PORT} \
+    -use_statestore -state_store_host=${IMPALA_STATE_STORE_HOST} \
+    -be_port=${IMPALA_BACKEND_PORT} -mem_limit=70%}</codeblock>
+            <p>
+              You can specify the memory limit using absolute notation such as
+              <codeph>500m</codeph> or <codeph>2G</codeph>, or as a percentage of physical
+              memory such as <codeph>60%</codeph>.
+            </p>
+
+            <note>
+              Queries that exceed the specified memory limit are aborted. Percentage limits are
+              based on the physical memory of the machine and do not consider cgroups.
+            </note>
+          </li>
+
+          <li>
+            Core dump enablement. To enable core dumps, change:
+<codeblock>export ENABLE_CORE_DUMPS=${ENABLE_COREDUMPS:-false}</codeblock>
+            <p>
+              to:
+            </p>
+<codeblock>export ENABLE_CORE_DUMPS=${ENABLE_COREDUMPS:-true}</codeblock>
+            <note>
+              The location of core dump files may vary according to your operating system
+              configuration. Other security settings may prevent Impala from writing core dumps
+              even when this option is enabled.
+            </note>
+          </li>
+
+          <li>
+            Authorization using the open source Sentry plugin. Specify the
+            <codeph>-server_name</codeph> and <codeph>-authorization_policy_file</codeph>
+            options as part of the <codeph>IMPALA_SERVER_ARGS</codeph> and
+            <codeph>IMPALA_STATE_STORE_ARGS</codeph> settings to enable the core Impala support
+            for authentication. See <xref href="impala_authorization.xml#secure_startup"/> for
+            details.
+          </li>
+
+          <li>
+            Auditing for successful or blocked Impala queries, another aspect of security.
+            Specify the <codeph>-audit_event_log_dir=<varname>directory_path</varname></codeph>
+            option and optionally the
+            <codeph>-max_audit_event_log_file_size=<varname>number_of_queries</varname></codeph>
+            and <codeph>-abort_on_failed_audit_event</codeph> options as part of the
+            <codeph>IMPALA_SERVER_ARGS</codeph> settings, for each Impala node, to enable and
+            customize auditing. See <xref href="impala_auditing.xml#auditing"/> for details.
+          </li>
+
+          <li>
+            Password protection for the Impala web UI, which listens on port 25000 by default.
+            This feature involves adding some or all of the
+            <codeph>--webserver_password_file</codeph>,
+            <codeph>--webserver_authentication_domain</codeph>, and
+            <codeph>--webserver_certificate_file</codeph> options to the
+            <codeph>IMPALA_SERVER_ARGS</codeph> and <codeph>IMPALA_STATE_STORE_ARGS</codeph>
+            settings. See <xref href="impala_security_webui.xml"/> for details.
+          </li>
+
+          <li id="default_query_options">
+            Another setting you might add to <codeph>IMPALA_SERVER_ARGS</codeph> is:
+<codeblock>-default_query_options='<varname>option</varname>=<varname>value</varname>,<varname>option</varname>=<varname>value</varname>,...'
+</codeblock>
+            These options control the behavior of queries performed by this
+            <cmdname>impalad</cmdname> instance. The option values you specify here override the
+            default values for <xref href="impala_query_options.xml#query_options">Impala query
+            options</xref>, as shown by the <codeph>SET</codeph> statement in
+            <cmdname>impala-shell</cmdname>.
+          </li>
+
+<!-- Removing this reference now that the options are de-emphasized / desupported in CDH 5.5 / Impala 2.3 and up.
+          <li rev="1.2">
+            Options for resource management, in conjunction with the YARN component. These options
+            include <codeph>-enable_rm</codeph> and <codeph>-cgroup_hierarchy_path</codeph>.
+            <ph rev="1.4.0">Additional options to help fine-tune the resource estimates are
+            <codeph>-\u2014rm_always_use_defaults</codeph>,
+            <codeph>-\u2014rm_default_memory=<varname>size</varname></codeph>, and
+            <codeph>-\u2014rm_default_cpu_cores</codeph>.</ph> For details about these options, see
+            <xref href="impala_resource_management.xml#rm_options"/>. See
+            <xref href="impala_resource_management.xml#resource_management"/> for information about resource
+            management in general.
+          </li>
+-->
+
+          <li>
+            During troubleshooting, <keyword keyref="support_org"/> might direct you to change other values,
+            particularly for <codeph>IMPALA_SERVER_ARGS</codeph>, to work around issues or
+            gather debugging information.
+          </li>
+        </ul>
+
+<!-- Removing this reference now that the options are de-emphasized / desupported in CDH 5.5 / Impala 2.3 and up.
+        <p conref="impala_resource_management.xml#rm_options/resource_management_impalad_options"/>
+-->
+
+        <note>
+          <p>
+            These startup options for the <cmdname>impalad</cmdname> daemon are different from
+            the command-line options for the <cmdname>impala-shell</cmdname> command. For the
+            <cmdname>impala-shell</cmdname> options, see
+            <xref href="impala_shell_options.xml#shell_options"/>.
+          </p>
+        </note>
+
+      </conbody>
+
+    </concept>
+
+    <concept audience="Cloudera" id="config_options_statestored_details">
+
+      <title>Configuration Options for statestored Daemon</title>
+
+      <conbody>
+
+        <p></p>
+
+      </conbody>
+
+    </concept>
+
+    <concept audience="Cloudera" id="config_options_catalogd_details">
+
+      <title>Configuration Options for catalogd Daemon</title>
+
+      <conbody>
+
+        <p></p>
+
+      </conbody>
+
+    </concept>
+
+  </concept>
+
+  <concept id="config_options_checking">
+
+    <title>Checking the Values of Impala Configuration Options</title>
+
+    <conbody>
+
+      <p>
+        You can check the current runtime value of all these settings through the Impala web
+        interface, available by default at
+        <codeph>http://<varname>impala_hostname</varname>:25000/varz</codeph> for the
+        <cmdname>impalad</cmdname> daemon,
+        <codeph>http://<varname>impala_hostname</varname>:25010/varz</codeph> for the
+        <cmdname>statestored</cmdname> daemon, or
+        <codeph>http://<varname>impala_hostname</varname>:25020/varz</codeph> for the
+        <cmdname>catalogd</cmdname> daemon. In the Cloudera Manager interface, you can see the
+        link to the appropriate <uicontrol><varname>service_name</varname> Web UI</uicontrol>
+        page when you look at the status page for a specific daemon on a specific host.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="config_options_impalad">
+
+    <title>Startup Options for impalad Daemon</title>
+
+    <conbody>
+
+      <p>
+        The <codeph>impalad</codeph> daemon implements the main Impala service, which performs
+        query processing and reads and writes the data files.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="config_options_statestored">
+
+    <title>Startup Options for statestored Daemon</title>
+
+    <conbody>
+
+      <p>
+        The <cmdname>statestored</cmdname> daemon implements the Impala statestore service,
+        which monitors the availability of Impala services across the cluster, and handles
+        situations such as nodes becoming unavailable or becoming available again.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="1.2" id="config_options_catalogd">
+
+    <title>Startup Options for catalogd Daemon</title>
+
+    <conbody>
+
+      <p>
+        The <cmdname>catalogd</cmdname> daemon implements the Impala catalog service, which
+        broadcasts metadata changes to all the Impala nodes when Impala creates a table, inserts
+        data, or performs other kinds of DDL and DML operations.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/load_catalog_in_background"/>
+
+    </conbody>
+
+  </concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_config_performance.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_config_performance.xml b/docs/topics/impala_config_performance.xml
new file mode 100644
index 0000000..0a44a33
--- /dev/null
+++ b/docs/topics/impala_config_performance.xml
@@ -0,0 +1,179 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="config_performance">
+
+  <title>Post-Installation Configuration for Impala</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Configuring"/>
+      <data name="Category" value="Administrators"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p id="p_24">
+      This section describes the mandatory and recommended configuration settings for Impala. If Impala is
+      installed using Cloudera Manager, some of these configurations are completed automatically; you must still
+      configure short-circuit reads manually. If you installed Impala without Cloudera Manager, or if you want to
+      customize your environment, consider making the changes described in this topic.
+    </p>
+
+    <p>
+<!-- Could conref this paragraph from ciiu_install.xml. -->
+      In some cases, depending on the level of Impala, CDH, and Cloudera Manager, you might need to add particular
+      component configuration details in one of the free-form fields on the Impala configuration pages within
+      Cloudera Manager. <ph conref="../shared/impala_common.xml#common/safety_valve"/>
+    </p>
+
+    <ul>
+      <li>
+        You must enable short-circuit reads, whether or not Impala was installed through Cloudera Manager. This
+        setting goes in the Impala configuration settings, not the Hadoop-wide settings.
+      </li>
+
+      <li>
+        If you installed Impala in an environment that is not managed by Cloudera Manager, you must enable block
+        location tracking, and you can optionally enable native checksumming for optimal performance.
+      </li>
+
+      <li>
+        If you deployed Impala using Cloudera Manager see
+        <xref href="impala_perf_testing.xml#performance_testing"/> to confirm proper configuration.
+      </li>
+    </ul>
+
+    <section id="section_fhq_wyv_ls">
+      <title>Mandatory: Short-Circuit Reads</title>
+      <p> Enabling short-circuit reads allows Impala to read local data directly
+        from the file system. This removes the need to communicate through the
+        DataNodes, improving performance. This setting also minimizes the number
+        of additional copies of data. Short-circuit reads requires
+          <codeph>libhadoop.so</codeph>
+        <!-- This link went stale. Not obvious how to keep it in sync with whatever Hadoop CDH is using behind the scenes. So hide the link for now. -->
+        <!--        (the <xref href="http://hadoop.apache.org/docs/r0.19.1/native_libraries.html" scope="external" format="html">Hadoop Native Library</xref>) -->
+        (the Hadoop Native Library) to be accessible to both the server and the
+        client. <codeph>libhadoop.so</codeph> is not available if you have
+        installed from a tarball. You must install from an
+        <codeph>.rpm</codeph>, <codeph>.deb</codeph>, or parcel to use
+        short-circuit local reads. <note> If you use Cloudera Manager, you can
+          enable short-circuit reads through a checkbox in the user interface
+          and that setting takes effect for Impala as well. </note>
+      </p>
+      <p>
+        <b>To configure DataNodes for short-circuit reads:</b>
+      </p>
+      <ol id="ol_qlq_wyv_ls">
+        <li id="copy_config_files"> Copy the client
+            <codeph>core-site.xml</codeph> and <codeph>hdfs-site.xml</codeph>
+          configuration files from the Hadoop configuration directory to the
+          Impala configuration directory. The default Impala configuration
+          location is <codeph>/etc/impala/conf</codeph>. </li>
+        <li>
+          <indexterm audience="Cloudera"
+            >dfs.client.read.shortcircuit</indexterm>
+          <indexterm audience="Cloudera">dfs.domain.socket.path</indexterm>
+          <indexterm audience="Cloudera"
+            >dfs.client.file-block-storage-locations.timeout.millis</indexterm>
+          On all Impala nodes, configure the following properties in <!-- Exact timing is unclear, since we say farther down to copy /etc/hadoop/conf/hdfs-site.xml to /etc/impala/conf.
+     Which wouldn't work if we already modified the Impala version of the file here. Not to mention that this
+     doesn't take the CM interface into account, where these /etc files might not exist in those locations. -->
+          <!--          <codeph>/etc/impala/conf/hdfs-site.xml</codeph> as shown: -->
+          Impala's copy of <codeph>hdfs-site.xml</codeph> as shown: <codeblock>&lt;property&gt;
+    &lt;name&gt;dfs.client.read.shortcircuit&lt;/name&gt;
+    &lt;value&gt;true&lt;/value&gt;
+&lt;/property&gt;
+
+&lt;property&gt;
+    &lt;name&gt;dfs.domain.socket.path&lt;/name&gt;
+    &lt;value&gt;/var/run/hdfs-sockets/dn&lt;/value&gt;
+&lt;/property&gt;
+
+&lt;property&gt;
+    &lt;name&gt;dfs.client.file-block-storage-locations.timeout.millis&lt;/name&gt;
+    &lt;value&gt;10000&lt;/value&gt;
+&lt;/property&gt;</codeblock>
+          <!-- Former socket.path value:    &lt;value&gt;/var/run/hadoop-hdfs/dn._PORT&lt;/value&gt; -->
+          <!--
+          <note>
+            The text <codeph>_PORT</codeph> appears just as shown; you do not need to
+            substitute a number.
+          </note>
+-->
+        </li>
+        <li>
+          <p> If <codeph>/var/run/hadoop-hdfs/</codeph> is group-writable, make
+            sure its group is <codeph>root</codeph>. </p>
+          <note> If you are also going to enable block location tracking, you
+            can skip copying configuration files and restarting DataNodes and go
+            straight to <xref href="#config_performance/block_location_tracking"
+             >Optional: Block Location Tracking</xref>.
+            Configuring short-circuit reads and block location tracking require
+            the same process of copying files and restarting services, so you
+            can complete that process once when you have completed all
+            configuration changes. Whether you copy files and restart services
+            now or during configuring block location tracking, short-circuit
+            reads are not enabled until you complete those final steps. </note>
+        </li>
+        <li id="restart_all_datanodes"> After applying these changes, restart
+          all DataNodes. </li>
+      </ol>
+    </section>
+
+    <section id="block_location_tracking">
+
+      <title>Mandatory: Block Location Tracking</title>
+
+      <p>
+        Enabling block location metadata allows Impala to know which disk data blocks are located on, allowing
+        better utilization of the underlying disks. Impala will not start unless this setting is enabled.
+      </p>
+
+      <p>
+        <b>To enable block location tracking:</b>
+      </p>
+
+      <ol>
+        <li>
+          For each DataNode, adding the following to the�<codeph>hdfs-site.xml</codeph> file:
+<codeblock>&lt;property&gt;
+  &lt;name&gt;dfs.datanode.hdfs-blocks-metadata.enabled&lt;/name&gt;
+  &lt;value&gt;true&lt;/value&gt;
+&lt;/property&gt; </codeblock>
+        </li>
+
+        <li conref="#config_performance/copy_config_files"/>
+
+        <li conref="#config_performance/restart_all_datanodes"/>
+      </ol>
+    </section>
+
+    <section id="native_checksumming">
+
+      <title>Optional: Native Checksumming</title>
+
+      <p>
+        Enabling native checksumming causes Impala to use an optimized native library for computing checksums, if
+        that library is available.
+      </p>
+
+      <p id="p_29">
+        <b>To enable native checksumming:</b>
+      </p>
+
+      <p>
+        If you installed CDH from packages, the native checksumming library is installed and setup correctly. In
+        such a case, no additional steps are required. Conversely, if you installed by other means, such as with
+        tarballs, native checksumming may not be available due to missing shared objects. Finding the message
+        "<codeph>Unable to load native-hadoop library for your platform... using builtin-java classes where
+        applicable</codeph>" in the Impala logs indicates native checksumming may be unavailable. To enable native
+        checksumming, you must build and install <codeph>libhadoop.so</codeph> (the
+        <!-- Another instance of stale link. -->
+        <!-- <xref href="http://hadoop.apache.org/docs/r0.19.1/native_libraries.html" scope="external" format="html">Hadoop Native Library</xref>). -->
+        Hadoop Native Library).
+      </p>
+    </section>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_connecting.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_connecting.xml b/docs/topics/impala_connecting.xml
new file mode 100644
index 0000000..354e698
--- /dev/null
+++ b/docs/topics/impala_connecting.xml
@@ -0,0 +1,202 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="connecting">
+
+  <title>Connecting to impalad through impala-shell</title>
+  <titlealts audience="PDF"><navtitle>Connecting to impalad</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="impala-shell"/>
+      <data name="Category" value="Network"/>
+      <data name="Category" value="DataNode"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+<!--
+TK: This would be a good theme for a tutorial topic.
+Lots of nuances to illustrate through sample code.
+-->
+
+    <p>
+      Within an <cmdname>impala-shell</cmdname> session, you can only issue queries while connected to an instance
+      of the <cmdname>impalad</cmdname> daemon. You can specify the connection information:
+      <ul>
+        <li>
+          Through command-line options when you run the <cmdname>impala-shell</cmdname> command.
+        </li>
+        <li>
+          Through a configuration file that is read when you run the <cmdname>impala-shell</cmdname> command.
+        </li>
+        <li>
+          During an <cmdname>impala-shell</cmdname> session, by issuing a <codeph>CONNECT</codeph> command.
+        </li>
+      </ul>
+      See <xref href="impala_shell_options.xml"/> for the command-line and configuration file options you can use.
+    </p>
+
+    <p>
+      You can connect to any DataNode where an instance of <cmdname>impalad</cmdname> is running,
+      and that host coordinates the execution of all queries sent to it.
+    </p>
+
+    <p>
+      For simplicity during development, you might always connect to the same host, perhaps running <cmdname>impala-shell</cmdname> on
+      the same host as <cmdname>impalad</cmdname> and specifying the hostname as <codeph>localhost</codeph>.
+    </p>
+
+    <p>
+      In a production environment, you might enable load balancing, in which you connect to specific host/port combination
+      but queries are forwarded to arbitrary hosts. This technique spreads the overhead of acting as the coordinator
+      node among all the DataNodes in the cluster. See <xref href="impala_proxy.xml"/> for details.
+    </p>
+
+    <p>
+      <b>To connect the Impala shell during shell startup:</b>
+    </p>
+
+    <ol>
+      <li>
+        Locate the hostname of a DataNode within the cluster that is running an instance of the
+        <cmdname>impalad</cmdname> daemon. If that DataNode uses a non-default port (something
+        other than port 21000) for <cmdname>impala-shell</cmdname> connections, find out the
+        port number also.
+      </li>
+
+      <li>
+        Use the <codeph>-i</codeph> option to the
+        <cmdname>impala-shell</cmdname> interpreter to specify the connection information for
+        that instance of <cmdname>impalad</cmdname>:
+<codeblock>
+# When you are logged into the same machine running impalad.
+# The prompt will reflect the current hostname.
+$ impala-shell
+
+# When you are logged into the same machine running impalad.
+# The host will reflect the hostname 'localhost'.
+$ impala-shell -i localhost
+
+# When you are logged onto a different host, perhaps a client machine
+# outside the Hadoop cluster.
+$ impala-shell -i <varname>some.other.hostname</varname>
+
+# When you are logged onto a different host, and impalad is listening
+# on a non-default port. Perhaps a load balancer is forwarding requests
+# to a different host/port combination behind the scenes.
+$ impala-shell -i <varname>some.other.hostname</varname>:<varname>port_number</varname>
+</codeblock>
+      </li>
+    </ol>
+
+    <p>
+      <b>To connect the Impala shell after shell startup:</b>
+    </p>
+
+    <ol>
+      <li>
+        Start the Impala shell with no connection:
+<codeblock>$ impala-shell</codeblock>
+        <p>
+          You should see a prompt like the following:
+        </p>
+<codeblock>Welcome to the Impala shell. Press TAB twice to see a list of available commands.
+
+Copyright (c) <varname>year</varname> Cloudera, Inc. All rights reserved.
+
+<ph conref="../shared/ImpalaVariables.xml#impala_vars/ShellBanner"/>
+[Not connected] &gt; </codeblock>
+      </li>
+
+      <li>
+        Locate the hostname of a DataNode within the cluster that is running an instance of the
+        <cmdname>impalad</cmdname> daemon. If that DataNode uses a non-default port (something
+        other than port 21000) for <cmdname>impala-shell</cmdname> connections, find out the
+        port number also.
+      </li>
+
+      <li>
+        Use the <codeph>connect</codeph> command to connect to an Impala instance. Enter a command of the form:
+<codeblock>[Not connected] &gt; connect <varname>impalad-host</varname>
+[<varname>impalad-host</varname>:21000] &gt;</codeblock>
+        <note>
+          Replace <varname>impalad-host</varname> with the hostname you have configured for any DataNode running
+          Impala in your environment. The changed prompt indicates a successful connection.
+        </note>
+      </li>
+    </ol>
+
+    <p>
+      <b>To start <cmdname>impala-shell</cmdname> in a specific database:</b>
+    </p>
+
+    <p>
+      You can use all the same connection options as in previous examples.
+      For simplicity, these examples assume that you are logged into one of
+      the DataNodes that is running the <cmdname>impalad</cmdname> daemon.
+    </p>
+
+    <ol>
+      <li>
+        Find the name of the database containing the relevant tables, views, and so
+        on that you want to operate on.
+      </li>
+
+      <li>
+        Use the <codeph>-d</codeph> option to the
+        <cmdname>impala-shell</cmdname> interpreter to connect and immediately
+        switch to the specified database, without the need for a <codeph>USE</codeph>
+        statement or fully qualified names:
+<codeblock>
+# Subsequent queries with unqualified names operate on
+# tables, views, and so on inside the database named 'staging'.
+$ impala-shell -i localhost -d staging
+
+# It is common during development, ETL, benchmarking, and so on
+# to have different databases containing the same table names
+# but with different contents or layouts.
+$ impala-shell -i localhost -d parquet_snappy_compression
+$ impala-shell -i localhost -d parquet_gzip_compression
+</codeblock>
+      </li>
+    </ol>
+
+    <p>
+      <b>To run one or several statements in non-interactive mode:</b>
+    </p>
+
+    <p>
+      You can use all the same connection options as in previous examples.
+      For simplicity, these examples assume that you are logged into one of
+      the DataNodes that is running the <cmdname>impalad</cmdname> daemon.
+    </p>
+
+    <ol>
+      <li>
+        Construct a statement, or a file containing a sequence of statements,
+        that you want to run in an automated way, without typing or copying
+        and pasting each time.
+      </li>
+
+      <li>
+        Invoke <cmdname>impala-shell</cmdname> with the <codeph>-q</codeph> option to run a single statement, or
+        the <codeph>-f</codeph> option to run a sequence of statements from a file.
+        The <cmdname>impala-shell</cmdname> command returns immediately, without going into
+        the interactive interpreter.
+<codeblock>
+# A utility command that you might run while developing shell scripts
+# to manipulate HDFS files.
+$ impala-shell -i localhost -d database_of_interest -q 'show tables'
+
+# A sequence of CREATE TABLE, CREATE VIEW, and similar DDL statements
+# can go into a file to make the setup process repeatable.
+$ impala-shell -i localhost -d database_of_interest -f recreate_tables.sql
+</codeblock>
+      </li>
+    </ol>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_conversion_functions.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_conversion_functions.xml b/docs/topics/impala_conversion_functions.xml
new file mode 100644
index 0000000..36f5ec4
--- /dev/null
+++ b/docs/topics/impala_conversion_functions.xml
@@ -0,0 +1,758 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="conversion_functions">
+
+  <title>Impala Type Conversion Functions</title>
+  <titlealts audience="PDF"><navtitle>Type Conversion Functions</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="Impala Data Types"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Querying"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Conversion functions are usually used in combination with other functions, to explicitly pass the expected
+      data types. Impala has strict rules regarding data types for function parameters. For example, Impala does
+      not automatically convert a <codeph>DOUBLE</codeph> value to <codeph>FLOAT</codeph>, a
+      <codeph>BIGINT</codeph> value to <codeph>INT</codeph>, or other conversion where precision could be lost or
+      overflow could occur. Also, for reporting or dealing with loosely defined schemas in big data contexts,
+      you might frequently need to convert values to or from the <codeph>STRING</codeph> type.
+    </p>
+
+    <note>
+      Although in CDH 5.5.0, the <codeph>SHOW FUNCTIONS</codeph> output for
+      database <codeph>_IMPALA_BUILTINS</codeph> contains some function signatures
+      matching the pattern <codeph>castto*</codeph>, these functions are not intended
+      for public use and are expected to be hidden in future.
+    </note>
+
+    <p>
+      <b>Function reference:</b>
+    </p>
+
+    <p>
+      Impala supports the following type conversion functions:
+    </p>
+
+<dl>
+
+<dlentry id="cast">
+<dt>
+<codeph>cast(<varname>expr</varname> AS <varname>type</varname>)</codeph>
+</dt>
+
+<dd>
+<indexterm audience="Cloudera">cast() function</indexterm>
+<b>Purpose:</b> Converts the value of an expression to any other type.
+If the expression value is of a type that cannot be converted to the target type, the result is <codeph>NULL</codeph>.
+<p><b>Usage notes:</b>
+Use <codeph>CAST</codeph> when passing a column value or literal to a function that
+expects a parameter with a different type.
+Frequently used in SQL operations such as <codeph>CREATE TABLE AS SELECT</codeph>
+and <codeph>INSERT ... VALUES</codeph> to ensure that values from various sources
+are of the appropriate type for the destination columns.
+Where practical, do a one-time <codeph>CAST()</codeph> operation during the ingestion process
+to make each column into the appropriate type, rather than using many <codeph>CAST()</codeph>
+operations in each query; doing type conversions for each row during each query can be expensive
+for tables with millions or billions of rows.
+</p>
+    <p conref="../shared/impala_common.xml#common/timezone_conversion_caveat"/>
+
+<p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>select concat('Here are the first ',10,' results.'); -- Fails
+select concat('Here are the first ',cast(10 as string),' results.'); -- Succeeds
+</codeblock>
+<p>
+The following example starts with a text table where every column has a type of <codeph>STRING</codeph>,
+which might be how you ingest data of unknown schema until you can verify the cleanliness of the underly values.
+Then it uses <codeph>CAST()</codeph> to create a new Parquet table with the same data, but using specific
+numeric data types for the columns with numeric data. Using numeric types of appropriate sizes can result in
+substantial space savings on disk and in memory, and performance improvements in queries,
+over using strings or larger-than-necessary numeric types.
+</p>
+<codeblock>create table t1 (name string, x string, y string, z string);
+
+create table t2 stored as parquet
+as select
+  name,
+  cast(x as bigint) x,
+  cast(y as timestamp) y,
+  cast(z as smallint) z
+from t1;
+
+describe t2;
++------+----------+---------+
+| name | type     | comment |
++------+----------+---------+
+| name | string   |         |
+| x    | bigint   |         |
+| y    | smallint |         |
+| z    | tinyint  |         |
++------+----------+---------+
+</codeblock>
+<p conref="../shared/impala_common.xml#common/related_info"/>
+<p>
+<!-- TK: Can you cast to or from MAP, ARRAY, STRUCT? -->
+  For details of casts from each kind of data type, see the description of
+  the appropriate type:
+  <xref href="impala_tinyint.xml#tinyint"/>,
+  <xref href="impala_smallint.xml#smallint"/>,
+  <xref href="impala_int.xml#int"/>,
+  <xref href="impala_bigint.xml#bigint"/>,
+  <xref href="impala_float.xml#float"/>,
+  <xref href="impala_double.xml#double"/>,
+  <xref href="impala_decimal.xml#decimal"/>,
+  <xref href="impala_string.xml#string"/>,
+  <xref href="impala_char.xml#char"/>,
+  <xref href="impala_varchar.xml#varchar"/>,
+  <xref href="impala_timestamp.xml#timestamp"/>,
+  <xref href="impala_boolean.xml#boolean"/>
+</p>
+</dd>
+</dlentry>
+
+<dlentry rev="2.3.0" id="casttobigint" audience="Cloudera">
+<dt>
+<codeph>casttobigint(type value)</codeph>
+</dt>
+<dd>
+<indexterm audience="Cloudera">casttobigint() function</indexterm>
+<b>Purpose:</b> Converts the value of an expression to <codeph>BIGINT</codeph>. If the expression value is of a type that cannot be converted to the target type, the result is <codeph>NULL</codeph>.
+<p><b>Return type:</b> <codeph>bigint</codeph></p>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_usage"/>
+<p conref="../shared/impala_common.xml#common/added_in_230"/>
+<p conref="../shared/impala_common.xml#common/example_blurb"/>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_example"/>
+<codeblock>create table small_types (x tinyint, y smallint, z int);
+
+create table big_types as
+  select casttobigint(x) as x, casttobigint(y) as y, casttobigint(z) as z
+    from small_types;
+
+describe big_types;
++------+--------+---------+
+| name | type   | comment |
++------+--------+---------+
+| x    | bigint |         |
+| y    | bigint |         |
+| z    | bigint |         |
++------+--------+---------+
+</codeblock>
+</dd>
+</dlentry>
+
+<dlentry rev="2.3.0" id="casttoboolean" audience="Cloudera">
+<dt>
+<codeph>casttoboolean(type value)</codeph>
+</dt>
+<dd>
+<indexterm audience="Cloudera">casttoboolean() function</indexterm>
+<b>Purpose:</b> Converts the value of an expression to <codeph>BOOLEAN</codeph>.
+Numeric values of 0 evaluate to <codeph>false</codeph>, and non-zero values evaluate to <codeph>true</codeph>.
+If the expression value is of a type that cannot be converted to the target type, the result is <codeph>NULL</codeph>.
+In particular, <codeph>STRING</codeph> values (even <codeph>'1'</codeph>, <codeph>'0'</codeph>, <codeph>'true'</codeph>
+or <codeph>'false'</codeph>) always return <codeph>NULL</codeph> when converted to <codeph>BOOLEAN</codeph>.
+<p><b>Return type:</b> <codeph>boolean</codeph></p>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_usage"/>
+<p conref="../shared/impala_common.xml#common/added_in_230"/>
+<p conref="../shared/impala_common.xml#common/example_blurb"/>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_example"/>
+<codeblock>select casttoboolean(0);
++------------------+
+| casttoboolean(0) |
++------------------+
+| false            |
++------------------+
+
+select casttoboolean(1);
++------------------+
+| casttoboolean(1) |
++------------------+
+| true             |
++------------------+
+
+select casttoboolean(99);
++-------------------+
+| casttoboolean(99) |
++-------------------+
+| true              |
++-------------------+
+
+select casttoboolean(0.0);
++--------------------+
+| casttoboolean(0.0) |
++--------------------+
+| false              |
++--------------------+
+
+select casttoboolean(0.5);
++--------------------+
+| casttoboolean(0.5) |
++--------------------+
+| true               |
++--------------------+
+
+select casttoboolean('');
++-------------------+
+| casttoboolean('') |
++-------------------+
+| NULL              |
++-------------------+
+
+select casttoboolean('yes');
++----------------------+
+| casttoboolean('yes') |
++----------------------+
+| NULL                 |
++----------------------+
+
+select casttoboolean('0');
++--------------------+
+| casttoboolean('0') |
++--------------------+
+| NULL               |
++--------------------+
+
+select casttoboolean('true');
++-----------------------+
+| casttoboolean('true') |
++-----------------------+
+| NULL                  |
++-----------------------+
+
+select casttoboolean('false');
++------------------------+
+| casttoboolean('false') |
++------------------------+
+| NULL                   |
++------------------------+
+</codeblock>
+</dd>
+</dlentry>
+
+<dlentry rev="2.3.0" id="casttochar" audience="Cloudera">
+<dt>
+<codeph>casttochar(type value)</codeph>
+</dt>
+<dd>
+<indexterm audience="Cloudera">casttochar() function</indexterm>
+<b>Purpose:</b> Converts the value of an expression to <codeph>CHAR</codeph>. If the expression value is of a type that cannot be converted to the target type, the result is <codeph>NULL</codeph>.
+<p><b>Return type:</b> <codeph>char</codeph></p>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_usage"/>
+<p conref="../shared/impala_common.xml#common/added_in_230"/>
+<p conref="../shared/impala_common.xml#common/example_blurb"/>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_example"/>
+<codeblock>create table char_types as select casttochar('hello world') as c1, casttochar('xyz') as c2, casttochar('x') as c3;
++-------------------+
+| summary           |
++-------------------+
+| Inserted 1 row(s) |
++-------------------+
+
+describe char_types;
++------+--------+---------+
+| name | type   | comment |
++------+--------+---------+
+| c1   | string |         |
+| c2   | string |         |
+| c3   | string |         |
++------+--------+---------+
+</codeblock>
+</dd>
+</dlentry>
+
+<dlentry rev="2.3.0" id="casttodecimal" audience="Cloudera">
+<dt>
+<codeph>casttodecimal(type value)</codeph>
+</dt>
+<dd>
+<indexterm audience="Cloudera">casttodecimal() function</indexterm>
+<b>Purpose:</b> Converts the value of an expression to <codeph>DECIMAL</codeph>. If the expression value is of a type that cannot be converted to the target type, the result is <codeph>NULL</codeph>.
+<p><b>Return type:</b> <codeph>decimal</codeph></p>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_usage"/>
+<p conref="../shared/impala_common.xml#common/added_in_230"/>
+<p conref="../shared/impala_common.xml#common/example_blurb"/>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_example"/>
+<codeblock>select casttodecimal(5.4);
++--------------------+
+| casttodecimal(5.4) |
++--------------------+
+| 5.4                |
++--------------------+
+</codeblock>
+</dd>
+</dlentry>
+
+<dlentry rev="2.3.0" id="casttodouble" audience="Cloudera">
+<dt>
+<codeph>casttodouble(type value)</codeph>
+</dt>
+<dd>
+<indexterm audience="Cloudera">casttodouble() function</indexterm>
+<b>Purpose:</b> Converts the value of an expression to <codeph>DOUBLE</codeph>. If the expression value is of a type that cannot be converted to the target type, the result is <codeph>NULL</codeph>.
+<p><b>Return type:</b> <codeph>double</codeph></p>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_usage"/>
+<p conref="../shared/impala_common.xml#common/added_in_230"/>
+<p conref="../shared/impala_common.xml#common/example_blurb"/>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_example"/>
+<codeblock>select casttodouble(5);
++-----------------+
+| casttodouble(5) |
++-----------------+
+| 5               |
++-----------------+
+
+select casttodouble('3.141');
++-----------------------+
+| casttodouble('3.141') |
++-----------------------+
+| 3.141                 |
++-----------------------+
+
+select casttodouble(1e6);
++--------------------+
+| casttodouble(1e+6) |
++--------------------+
+| 1000000            |
++--------------------+
+
+select casttodouble(true);
++--------------------+
+| casttodouble(true) |
++--------------------+
+| 1                  |
++--------------------+
+
+select casttodouble(now());
++---------------------+
+| casttodouble(now()) |
++---------------------+
+| 1447622306.031178   |
++---------------------+
+</codeblock>
+</dd>
+</dlentry>
+
+<dlentry rev="2.3.0" id="casttofloat" audience="Cloudera">
+<dt>
+<codeph>casttofloat(type value)</codeph>
+</dt>
+<dd>
+<indexterm audience="Cloudera">casttofloat() function</indexterm>
+<b>Purpose:</b> Converts the value of an expression to <codeph>FLOAT</codeph>. If the expression value is of a type that cannot be converted to the target type, the result is <codeph>NULL</codeph>.
+<p><b>Return type:</b> <codeph>float</codeph></p>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_usage"/>
+<p conref="../shared/impala_common.xml#common/added_in_230"/>
+<p conref="../shared/impala_common.xml#common/example_blurb"/>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_example"/>
+<codeblock>select casttofloat(5);
++----------------+
+| casttofloat(5) |
++----------------+
+| 5              |
++----------------+
+
+select casttofloat('3.141');
++----------------------+
+| casttofloat('3.141') |
++----------------------+
+| 3.141000032424927    |
++----------------------+
+
+select casttofloat(1e6);
++-------------------+
+| casttofloat(1e+6) |
++-------------------+
+| 1000000           |
++-------------------+
+
+select casttofloat(true);
++-------------------+
+| casttofloat(true) |
++-------------------+
+| 1                 |
++-------------------+
+
+select casttofloat(now());
++--------------------+
+| casttofloat(now()) |
++--------------------+
+| 1447622400         |
++--------------------+
+</codeblock>
+</dd>
+</dlentry>
+
+<dlentry rev="2.3.0" id="casttoint" audience="Cloudera">
+<dt>
+<codeph>casttoint(type value)</codeph>
+</dt>
+<dd>
+<indexterm audience="Cloudera">casttoint() function</indexterm>
+<b>Purpose:</b> Converts the value of an expression to <codeph>INT</codeph>. If the expression value is of a type that cannot be converted to the target type, the result is <codeph>NULL</codeph>.
+<p><b>Return type:</b> <codeph>int</codeph></p>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_usage"/>
+<p conref="../shared/impala_common.xml#common/added_in_230"/>
+<p conref="../shared/impala_common.xml#common/example_blurb"/>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_example"/>
+<codeblock>select casttoint(5.4);
++----------------+
+| casttoint(5.4) |
++----------------+
+| 5              |
++----------------+
+
+select casttoint(true);
++-----------------+
+| casttoint(true) |
++-----------------+
+| 1               |
++-----------------+
+
+select casttoint(now());
++------------------+
+| casttoint(now()) |
++------------------+
+| 1447622487       |
++------------------+
+
+select casttoint('3.141');
++--------------------+
+| casttoint('3.141') |
++--------------------+
+| NULL               |
++--------------------+
+
+select casttoint('3');
++----------------+
+| casttoint('3') |
++----------------+
+| 3              |
++----------------+
+</codeblock>
+</dd>
+</dlentry>
+
+<dlentry rev="2.3.0" id="casttosmallint" audience="Cloudera">
+<dt>
+<codeph>casttosmallint(type value)</codeph>
+</dt>
+<dd>
+<indexterm audience="Cloudera">casttosmallint() function</indexterm>
+<b>Purpose:</b> Converts the value of an expression to <codeph>SMALLINT</codeph>. If the expression value is of a type that cannot be converted to the target type, the result is <codeph>NULL</codeph>.
+<p><b>Return type:</b> <codeph>smallint</codeph></p>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_usage"/>
+<p conref="../shared/impala_common.xml#common/added_in_230"/>
+<p conref="../shared/impala_common.xml#common/example_blurb"/>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_example"/>
+<codeblock>create table big_types (x bigint, y int, z smallint);
+
+create table small_types as
+  select casttosmallint(x) as x, casttosmallint(y) as y, casttosmallint(z) as z
+    from big_types;
+
+describe small_types;
++------+----------+---------+
+| name | type     | comment |
++------+----------+---------+
+| x    | smallint |         |
+| y    | smallint |         |
+| z    | smallint |         |
++------+----------+---------+
+</codeblock>
+</dd>
+</dlentry>
+
+<dlentry rev="2.3.0" id="casttostring" audience="Cloudera">
+<dt>
+<codeph>casttostring(type value)</codeph>
+</dt>
+<dd>
+<indexterm audience="Cloudera">casttostring() function</indexterm>
+<b>Purpose:</b> Converts the value of an expression to <codeph>STRING</codeph>. If the expression value is of a type that cannot be converted to the target type, the result is <codeph>NULL</codeph>.
+<p><b>Return type:</b> <codeph>string</codeph></p>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_usage"/>
+<p conref="../shared/impala_common.xml#common/added_in_230"/>
+<p conref="../shared/impala_common.xml#common/example_blurb"/>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_example"/>
+<codeblock>create table numeric_types (x int, y bigint, z tinyint);
+
+create table string_types as
+  select casttostring(x) as x, casttostring(y) as y, casttostring(z) as z
+    from numeric_types;
+
+describe string_types;
++------+--------+---------+
+| name | type   | comment |
++------+--------+---------+
+| x    | string |         |
+| y    | string |         |
+| z    | string |         |
++------+--------+---------+
+</codeblock>
+</dd>
+</dlentry>
+
+<dlentry rev="2.3.0" id="casttotimestamp" audience="Cloudera">
+<dt>
+<codeph>casttotimestamp(type value)</codeph>
+</dt>
+<dd>
+<indexterm audience="Cloudera">casttotimestamp() function</indexterm>
+<b>Purpose:</b> Converts the value of an expression to <codeph>TIMESTAMP</codeph>. If the expression value is of a type that cannot be converted to the target type, the result is <codeph>NULL</codeph>.
+<p><b>Return type:</b> <codeph>timestamp</codeph></p>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_usage"/>
+<p conref="../shared/impala_common.xml#common/added_in_230"/>
+<p conref="../shared/impala_common.xml#common/example_blurb"/>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_example"/>
+<codeblock>select casttotimestamp(1000);
++-----------------------+
+| casttotimestamp(1000) |
++-----------------------+
+| 1970-01-01 00:16:40   |
++-----------------------+
+
+select casttotimestamp(1000.0);
++-------------------------+
+| casttotimestamp(1000.0) |
++-------------------------+
+| 1970-01-01 00:16:40     |
++-------------------------+
+
+select casttotimestamp('1000');
++-------------------------+
+| casttotimestamp('1000') |
++-------------------------+
+| NULL                    |
++-------------------------+
+</codeblock>
+</dd>
+</dlentry>
+
+<dlentry rev="2.3.0" id="casttotinyint" audience="Cloudera">
+<dt>
+<codeph>casttotinyint(type value)</codeph>
+</dt>
+<dd>
+<indexterm audience="Cloudera">casttotinyint() function</indexterm>
+<b>Purpose:</b> Converts the value of an expression to <codeph>TINYINT</codeph>. If the expression value is of a type that cannot be converted to the target type, the result is <codeph>NULL</codeph>.
+<p><b>Return type:</b> <codeph>tinyint</codeph></p>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_usage"/>
+<p conref="../shared/impala_common.xml#common/added_in_230"/>
+<p conref="../shared/impala_common.xml#common/example_blurb"/>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_example"/>
+<codeblock>create table big_types (x bigint, y int, z smallint);
+
+create table tiny_types as
+  select casttotinyint(x) as x, casttotinyint(y) as y, casttotinyint(z) as z
+    from big_types;
+
+describe tiny_types;
++------+---------+---------+
+| name | type    | comment |
++------+---------+---------+
+| x    | tinyint |         |
+| y    | tinyint |         |
+| z    | tinyint |         |
++------+---------+---------+
+</codeblock>
+</dd>
+</dlentry>
+
+<dlentry rev="2.3.0" id="casttovarchar" audience="Cloudera">
+<dt>
+<codeph>casttovarchar(type value)</codeph>
+</dt>
+<dd>
+<indexterm audience="Cloudera">casttovarchar() function</indexterm>
+<b>Purpose:</b> Converts the value of an expression to <codeph>VARCHAR</codeph>. If the expression value is of a type that cannot be converted to the target type, the result is <codeph>NULL</codeph>.
+<p><b>Return type:</b> <codeph>varchar</codeph></p>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_usage"/>
+<p conref="../shared/impala_common.xml#common/added_in_230"/>
+<p conref="../shared/impala_common.xml#common/example_blurb"/>
+<p conref="../shared/impala_common.xml#common/cast_convenience_fn_example"/>
+<codeblock>select casttovarchar('abcd');
++-----------------------+
+| casttovarchar('abcd') |
++-----------------------+
+| abcd                  |
++-----------------------+
+
+select casttovarchar(999);
++--------------------+
+| casttovarchar(999) |
++--------------------+
+| 999                |
++--------------------+
+
+select casttovarchar(999.5);
++----------------------+
+| casttovarchar(999.5) |
++----------------------+
+| 999.5                |
++----------------------+
+
+select casttovarchar(now());
++-------------------------------+
+| casttovarchar(now())          |
++-------------------------------+
+| 2015-11-15 21:26:13.528073000 |
++-------------------------------+
+
+select casttovarchar(true);
++---------------------+
+| casttovarchar(true) |
++---------------------+
+| 1                   |
++---------------------+
+</codeblock>
+</dd>
+</dlentry>
+
+<dlentry rev="2.3.0" id="typeof">
+<dt>
+<codeph>typeof(type value)</codeph>
+</dt>
+<dd>
+<indexterm audience="Cloudera">typeof() function</indexterm>
+<b>Purpose:</b> Returns the name of the data type corresponding to an expression. For types with
+extra attributes, such as length for <codeph>CHAR</codeph> and <codeph>VARCHAR</codeph>,
+or precision and scale for <codeph>DECIMAL</codeph>, includes the full specification of the type.
+<!-- To do: How about for columns of complex types? Or fields within complex types? -->
+<p><b>Return type:</b> <codeph>string</codeph></p>
+<p><b>Usage notes:</b> Typically used in interactive exploration of a schema, or in application code that programmatically generates schema definitions such as <codeph>CREATE TABLE</codeph> statements.
+For example, previously, to understand the type of an expression such as
+<codeph>col1 / col2</codeph> or <codeph>concat(col1, col2, col3)</codeph>,
+you might have created a dummy table with a single row, using syntax such as <codeph>CREATE TABLE foo AS SELECT 5 / 3.0</codeph>,
+and then doing a <codeph>DESCRIBE</codeph> to see the type of the row.
+Or you might have done a <codeph>CREATE TABLE AS SELECT</codeph> operation to create a table and
+copy data into it, only learning the types of the columns by doing a <codeph>DESCRIBE</codeph> afterward.
+This technique is especially useful for arithmetic expressions involving <codeph>DECIMAL</codeph> types,
+because the precision and scale of the result is typically different than that of the operands.
+</p>
+<p conref="../shared/impala_common.xml#common/added_in_230"/>
+<p conref="../shared/impala_common.xml#common/example_blurb"/>
+<p>
+These examples show how to check the type of a simple literal or function value.
+Notice how adding even tiny integers together changes the data type of the result to
+avoid overflow, and how the results of arithmetic operations on <codeph>DECIMAL</codeph> values
+have specific precision and scale attributes.
+</p>
+<codeblock>select typeof(2)
++-----------+
+| typeof(2) |
++-----------+
+| TINYINT   |
++-----------+
+
+select typeof(2+2)
++---------------+
+| typeof(2 + 2) |
++---------------+
+| SMALLINT      |
++---------------+
+
+select typeof('xyz')
++---------------+
+| typeof('xyz') |
++---------------+
+| STRING        |
++---------------+
+
+select typeof(now())
++---------------+
+| typeof(now()) |
++---------------+
+| TIMESTAMP     |
++---------------+
+
+select typeof(5.3 / 2.1)
++-------------------+
+| typeof(5.3 / 2.1) |
++-------------------+
+| DECIMAL(6,4)      |
++-------------------+
+
+select typeof(5.30001 / 2342.1);
++--------------------------+
+| typeof(5.30001 / 2342.1) |
++--------------------------+
+| DECIMAL(13,11)           |
++--------------------------+
+
+select typeof(typeof(2+2))
++-----------------------+
+| typeof(typeof(2 + 2)) |
++-----------------------+
+| STRING                |
++-----------------------+
+</codeblock>
+
+<p>
+This example shows how even if you do not have a record of the type of a column,
+for example because the type was changed by <codeph>ALTER TABLE</codeph> after the
+original <codeph>CREATE TABLE</codeph>, you can still find out the type in a
+more compact form than examining the full <codeph>DESCRIBE</codeph> output.
+Remember to use <codeph>LIMIT 1</codeph> in such cases, to avoid an identical
+result value for every row in the table.
+</p>
+<codeblock>create table typeof_example (a int, b tinyint, c smallint, d bigint);
+
+/* Empty result set if there is no data in the table. */
+select typeof(a) from typeof_example;
+
+/* OK, now we have some data but the type of column A is being changed. */
+insert into typeof_example values (1, 2, 3, 4);
+alter table typeof_example change a a bigint;
+
+/* We can always find out the current type of that column without doing a full DESCRIBE. */
+select typeof(a) from typeof_example limit 1;
++-----------+
+| typeof(a) |
++-----------+
+| BIGINT    |
++-----------+
+</codeblock>
+<p>
+This example shows how you might programmatically generate a <codeph>CREATE TABLE</codeph> statement
+with the appropriate column definitions to hold the result values of arbitrary expressions.
+The <codeph>typeof()</codeph> function lets you construct a detailed <codeph>CREATE TABLE</codeph> statement
+without actually creating the table, as opposed to <codeph>CREATE TABLE AS SELECT</codeph> operations
+where you create the destination table but only learn the column data types afterward through <codeph>DESCRIBE</codeph>.
+</p>
+<codeblock>describe typeof_example;
++------+----------+---------+
+| name | type     | comment |
++------+----------+---------+
+| a    | bigint   |         |
+| b    | tinyint  |         |
+| c    | smallint |         |
+| d    | bigint   |         |
++------+----------+---------+
+
+/* An ETL or business intelligence tool might create variations on a table with different file formats,
+   different sets of columns, and so on. TYPEOF() lets an application introspect the types of the original columns. */
+select concat('create table derived_table (a ', typeof(a), ', b ', typeof(b), ', c ',
+    typeof(c), ', d ', typeof(d), ') stored as parquet;')
+  as 'create table statement'
+from typeof_example limit 1;
++-------------------------------------------------------------------------------------------+
+| create table statement                                                                    |
++-------------------------------------------------------------------------------------------+
+| create table derived_table (a BIGINT, b TINYINT, c SMALLINT, d BIGINT) stored as parquet; |
++-------------------------------------------------------------------------------------------+
+</codeblock>
+</dd>
+</dlentry>
+
+</dl>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_count.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_count.xml b/docs/topics/impala_count.xml
new file mode 100644
index 0000000..4f9a697
--- /dev/null
+++ b/docs/topics/impala_count.xml
@@ -0,0 +1,236 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="count">
+
+  <title>COUNT Function</title>
+  <titlealts audience="PDF"><navtitle>COUNT</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="Analytic Functions"/>
+      <data name="Category" value="Aggregate Functions"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">count() function</indexterm>
+      An aggregate function that returns the number of rows, or the number of non-<codeph>NULL</codeph> rows.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>COUNT([DISTINCT | ALL] <varname>expression</varname>) [OVER (<varname>analytic_clause</varname>)]</codeblock>
+
+    <p>
+      Depending on the argument, <codeph>COUNT()</codeph> considers rows that meet certain conditions:
+    </p>
+
+    <ul>
+      <li>
+        The notation <codeph>COUNT(*)</codeph> includes <codeph>NULL</codeph> values in the total.
+      </li>
+
+      <li>
+        The notation <codeph>COUNT(<varname>column_name</varname>)</codeph> only considers rows where the column
+        contains a non-<codeph>NULL</codeph> value.
+      </li>
+
+      <li>
+        You can also combine <codeph>COUNT</codeph> with the <codeph>DISTINCT</codeph> operator to eliminate
+        duplicates before counting, and to count the combinations of values across multiple columns.
+      </li>
+    </ul>
+
+    <p>
+      When the query contains a <codeph>GROUP BY</codeph> clause, returns one value for each combination of
+      grouping values.
+    </p>
+
+    <p>
+      <b>Return type:</b> <codeph>BIGINT</codeph>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/partition_key_optimization"/>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_aggregation_explanation"/>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_aggregation_example"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>-- How many rows total are in the table, regardless of NULL values?
+select count(*) from t1;
+-- How many rows are in the table with non-NULL values for a column?
+select count(c1) from t1;
+-- Count the rows that meet certain conditions.
+-- Again, * includes NULLs, so COUNT(*) might be greater than COUNT(col).
+select count(*) from t1 where x &gt; 10;
+select count(c1) from t1 where x &gt; 10;
+-- Can also be used in combination with DISTINCT and/or GROUP BY.
+-- Combine COUNT and DISTINCT to find the number of unique values.
+-- Must use column names rather than * with COUNT(DISTINCT ...) syntax.
+-- Rows with NULL values are not counted.
+select count(distinct c1) from t1;
+-- Rows with a NULL value in _either_ column are not counted.
+select count(distinct c1, c2) from t1;
+-- Return more than one result.
+select month, year, count(distinct visitor_id) from web_stats group by month, year;
+</codeblock>
+
+    <p rev="2.0.0">
+      The following examples show how to use <codeph>COUNT()</codeph> in an analytic context. They use a table
+      containing integers from 1 to 10. Notice how the <codeph>COUNT()</codeph> is reported for each input value, as
+      opposed to the <codeph>GROUP BY</codeph> clause which condenses the result set.
+<codeblock>select x, property, count(x) over (partition by property) as count from int_t where property in ('odd','even');
++----+----------+-------+
+| x  | property | count |
++----+----------+-------+
+| 2  | even     | 5     |
+| 4  | even     | 5     |
+| 6  | even     | 5     |
+| 8  | even     | 5     |
+| 10 | even     | 5     |
+| 1  | odd      | 5     |
+| 3  | odd      | 5     |
+| 5  | odd      | 5     |
+| 7  | odd      | 5     |
+| 9  | odd      | 5     |
++----+----------+-------+
+</codeblock>
+
+Adding an <codeph>ORDER BY</codeph> clause lets you experiment with results that are cumulative or apply to a moving
+set of rows (the <q>window</q>). The following examples use <codeph>COUNT()</codeph> in an analytic context
+(that is, with an <codeph>OVER()</codeph> clause) to produce a running count of all the even values,
+then a running count of all the odd values. The basic <codeph>ORDER BY x</codeph> clause implicitly
+activates a window clause of <codeph>RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW</codeph>,
+which is effectively the same as <codeph>ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW</codeph>,
+therefore all of these examples produce the same results:
+<codeblock>select x, property,
+  count(x) over (partition by property <b>order by x</b>) as 'cumulative count'
+  from int_t where property in ('odd','even');
++----+----------+------------------+
+| x  | property | cumulative count |
++----+----------+------------------+
+| 2  | even     | 1                |
+| 4  | even     | 2                |
+| 6  | even     | 3                |
+| 8  | even     | 4                |
+| 10 | even     | 5                |
+| 1  | odd      | 1                |
+| 3  | odd      | 2                |
+| 5  | odd      | 3                |
+| 7  | odd      | 4                |
+| 9  | odd      | 5                |
++----+----------+------------------+
+
+select x, property,
+  count(x) over
+  (
+    partition by property
+    <b>order by x</b>
+    <b>range between unbounded preceding and current row</b>
+  ) as 'cumulative total'
+from int_t where property in ('odd','even');
++----+----------+------------------+
+| x  | property | cumulative count |
++----+----------+------------------+
+| 2  | even     | 1                |
+| 4  | even     | 2                |
+| 6  | even     | 3                |
+| 8  | even     | 4                |
+| 10 | even     | 5                |
+| 1  | odd      | 1                |
+| 3  | odd      | 2                |
+| 5  | odd      | 3                |
+| 7  | odd      | 4                |
+| 9  | odd      | 5                |
++----+----------+------------------+
+
+select x, property,
+  count(x) over
+  (
+    partition by property
+    <b>order by x</b>
+    <b>rows between unbounded preceding and current row</b>
+  ) as 'cumulative total'
+  from int_t where property in ('odd','even');
++----+----------+------------------+
+| x  | property | cumulative count |
++----+----------+------------------+
+| 2  | even     | 1                |
+| 4  | even     | 2                |
+| 6  | even     | 3                |
+| 8  | even     | 4                |
+| 10 | even     | 5                |
+| 1  | odd      | 1                |
+| 3  | odd      | 2                |
+| 5  | odd      | 3                |
+| 7  | odd      | 4                |
+| 9  | odd      | 5                |
++----+----------+------------------+
+</codeblock>
+
+The following examples show how to construct a moving window, with a running count taking into account 1 row before
+and 1 row after the current row, within the same partition (all the even values or all the odd values).
+Therefore, the count is consistently 3 for rows in the middle of the window, and 2 for
+rows near the ends of the window, where there is no preceding or no following row in the partition.
+Because of a restriction in the Impala <codeph>RANGE</codeph> syntax, this type of
+moving window is possible with the <codeph>ROWS BETWEEN</codeph> clause but not the <codeph>RANGE BETWEEN</codeph>
+clause:
+<codeblock>select x, property,
+  count(x) over
+  (
+    partition by property
+    <b>order by x</b>
+    <b>rows between 1 preceding and 1 following</b>
+  ) as 'moving total'
+  from int_t where property in ('odd','even');
++----+----------+--------------+
+| x  | property | moving total |
++----+----------+--------------+
+| 2  | even     | 2            |
+| 4  | even     | 3            |
+| 6  | even     | 3            |
+| 8  | even     | 3            |
+| 10 | even     | 2            |
+| 1  | odd      | 2            |
+| 3  | odd      | 3            |
+| 5  | odd      | 3            |
+| 7  | odd      | 3            |
+| 9  | odd      | 2            |
++----+----------+--------------+
+
+-- Doesn't work because of syntax restriction on RANGE clause.
+select x, property,
+  count(x) over
+  (
+    partition by property
+    <b>order by x</b>
+    <b>range between 1 preceding and 1 following</b>
+  ) as 'moving total'
+from int_t where property in ('odd','even');
+ERROR: AnalysisException: RANGE is only supported with both the lower and upper bounds UNBOUNDED or one UNBOUNDED and the other CURRENT ROW.
+</codeblock>
+    </p>
+
+    <note conref="../shared/impala_common.xml#common/multiple_count_distinct"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_analytic_functions.xml#analytic_functions"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_create_data_source.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_create_data_source.xml b/docs/topics/impala_create_data_source.xml
new file mode 100644
index 0000000..a93d6bc
--- /dev/null
+++ b/docs/topics/impala_create_data_source.xml
@@ -0,0 +1,35 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept audience="Cloudera" rev="1.4.0" id="create_data_source">
+
+  <title>CREATE DATA SOURCE Statement</title>
+  <titlealts audience="PDF"><navtitle>CREATE DATA SOURCE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">CREATE DATA SOURCE statement</indexterm>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/sync_ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_create_database.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_create_database.xml b/docs/topics/impala_create_database.xml
new file mode 100644
index 0000000..de556bc
--- /dev/null
+++ b/docs/topics/impala_create_database.xml
@@ -0,0 +1,137 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="create_database">
+
+  <title>CREATE DATABASE Statement</title>
+  <titlealts audience="PDF"><navtitle>CREATE DATABASE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Databases"/>
+      <data name="Category" value="Schemas"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="S3"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">CREATE DATABASE statement</indexterm>
+      Creates a new database.
+    </p>
+
+    <p>
+      In Impala, a database is both:
+    </p>
+
+    <ul>
+      <li>
+        A logical construct for grouping together related tables, views, and functions within their own namespace.
+        You might use a separate database for each application, set of related tables, or round of experimentation.
+      </li>
+
+      <li>
+        A physical construct represented by a directory tree in HDFS. Tables (internal tables), partitions, and
+        data files are all located under this directory. You can perform HDFS-level operations such as backing it up and measuring space usage,
+        or remove it with a <codeph>DROP DATABASE</codeph> statement.
+      </li>
+    </ul>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] <varname>database_name</varname>[COMMENT '<varname>database_comment</varname>']
+  [LOCATION <varname>hdfs_path</varname>];</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      A database is physically represented as a directory in HDFS, with a filename extension <codeph>.db</codeph>,
+      under the main Impala data directory. If the associated HDFS directory does not exist, it is created for you.
+      All databases and their associated directories are top-level objects, with no physical or logical nesting.
+    </p>
+
+    <p>
+      After creating a database, to make it the current database within an <cmdname>impala-shell</cmdname> session,
+      use the <codeph>USE</codeph> statement. You can refer to tables in the current database without prepending
+      any qualifier to their names.
+    </p>
+
+    <p>
+      When you first connect to Impala through <cmdname>impala-shell</cmdname>, the database you start in (before
+      issuing any <codeph>CREATE DATABASE</codeph> or <codeph>USE</codeph> statements) is named
+      <codeph>default</codeph>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/builtins_db"/>
+
+    <p>
+      After creating a database, your <cmdname>impala-shell</cmdname> session or another
+      <cmdname>impala-shell</cmdname> connected to the same node can immediately access that database. To access
+      the database through the Impala daemon on a different node, issue the <codeph>INVALIDATE METADATA</codeph>
+      statement first while connected to that other node.
+    </p>
+
+    <p>
+      Setting the <codeph>LOCATION</codeph> attribute for a new database is a way to work with sets of files in an
+      HDFS directory structure outside the default Impala data directory, as opposed to setting the
+      <codeph>LOCATION</codeph> attribute for each individual table.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/sync_ddl_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/hive_blurb"/>
+
+    <p>
+      When you create a database in Impala, the database can also be used by Hive.
+      When you create a database in Hive, issue an <codeph>INVALIDATE METADATA</codeph>
+      statement in Impala to make Impala permanently aware of the new database.
+    </p>
+
+    <p>
+      The <codeph>SHOW DATABASES</codeph> statement lists all databases, or the databases whose name
+      matches a wildcard pattern. <ph rev="2.5.0">In <keyword keyref="impala25_full"/> and higher, the
+      <codeph>SHOW DATABASES</codeph> output includes a second column that displays the associated
+      comment, if any, for each database.</ph>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/s3_blurb"/>
+
+    <p rev="2.6.0 CDH-39913 IMPALA-1878">
+      To specify that any tables created within a database reside on the Amazon S3 system,
+      you can include an <codeph>s3a://</codeph> prefix on the <codeph>LOCATION</codeph>
+      attribute. In <keyword keyref="impala26_full"/> and higher, Impala automatically creates any
+      required folders as the databases, tables, and partitions are created, and removes
+      them when they are dropped.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/s3_ddl"/>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+    <p rev="CDH-19187">
+      The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+      typically the <codeph>impala</codeph> user, must have write
+      permission for the parent HDFS directory under which the database
+      is located.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <codeblock conref="../shared/impala_common.xml#common/create_drop_db_example"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_databases.xml#databases"/>, <xref href="impala_drop_database.xml#drop_database"/>,
+      <xref href="impala_use.xml#use"/>, <xref href="impala_show.xml#show_databases"/>,
+      <xref href="impala_tables.xml#tables"/>
+    </p>
+  </conbody>
+</concept>



[15/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_partitioning.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_partitioning.xml b/docs/topics/impala_partitioning.xml
new file mode 100644
index 0000000..1aedfa1
--- /dev/null
+++ b/docs/topics/impala_partitioning.xml
@@ -0,0 +1,582 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="partitioning">
+
+  <title>Partitioning for Impala Tables</title>
+
+  <titlealts audience="PDF">
+
+    <navtitle>Partitioning</navtitle>
+
+  </titlealts>
+
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">partitioning</indexterm>
+      By default, all the data files for a table are located in a single directory. Partitioning is a technique for physically dividing the
+      data during loading, based on values from one or more columns, to speed up queries that test those columns. For example, with a
+      <codeph>school_records</codeph> table partitioned on a <codeph>year</codeph> column, there is a separate data directory for each
+      different year value, and all the data for that year is stored in a data file in that directory. A query that includes a
+      <codeph>WHERE</codeph> condition such as <codeph>YEAR=1966</codeph>, <codeph>YEAR IN (1989,1999)</codeph>, or <codeph>YEAR BETWEEN
+      1984 AND 1989</codeph> can examine only the data files from the appropriate directory or directories, greatly reducing the amount of
+      data to read and test.
+    </p>
+
+    <p outputclass="toc inpage"/>
+
+    <p>
+      See <xref href="impala_tutorial.xml#tut_external_partition_data"/> for an example that illustrates the syntax for creating partitioned
+      tables, the underlying directory structure in HDFS, and how to attach a partitioned Impala external table to data files stored
+      elsewhere in HDFS.
+    </p>
+
+    <p>
+      Parquet is a popular format for partitioned Impala tables because it is well suited to handle huge data volumes. See
+      <xref href="impala_parquet.xml#parquet_performance"/> for performance considerations for partitioned Parquet tables.
+    </p>
+
+    <p>
+      See <xref href="impala_literals.xml#null"/> for details about how <codeph>NULL</codeph> values are represented in partitioned tables.
+    </p>
+
+    <p rev="2.2.0">
+      See <xref href="impala_s3.xml#s3"/> for details about setting up tables where some or all partitions reside on the Amazon Simple
+      Storage Service (S3).
+    </p>
+
+  </conbody>
+
+  <concept id="partitioning_choosing">
+
+    <title>When to Use Partitioned Tables</title>
+
+    <conbody>
+
+      <p>
+        Partitioning is typically appropriate for:
+      </p>
+
+      <ul>
+        <li>
+          Tables that are very large, where reading the entire data set takes an impractical amount of time.
+        </li>
+
+        <li>
+          Tables that are always or almost always queried with conditions on the partitioning columns. In our example of a table partitioned
+          by year, <codeph>SELECT COUNT(*) FROM school_records WHERE year = 1985</codeph> is efficient, only examining a small fraction of
+          the data; but <codeph>SELECT COUNT(*) FROM school_records</codeph> has to process a separate data file for each year, resulting in
+          more overall work than in an unpartitioned table. You would probably not partition this way if you frequently queried the table
+          based on last name, student ID, and so on without testing the year.
+        </li>
+
+        <li>
+          Columns that have reasonable cardinality (number of different values). If a column only has a small number of values, for example
+          <codeph>Male</codeph> or <codeph>Female</codeph>, you do not gain much efficiency by eliminating only about 50% of the data to
+          read for each query. If a column has only a few rows matching each value, the number of directories to process can become a
+          limiting factor, and the data file in each directory could be too small to take advantage of the Hadoop mechanism for transmitting
+          data in multi-megabyte blocks. For example, you might partition census data by year, store sales data by year and month, and web
+          traffic data by year, month, and day. (Some users with high volumes of incoming data might even partition down to the individual
+          hour and minute.)
+        </li>
+
+        <li>
+          Data that already passes through an extract, transform, and load (ETL) pipeline. The values of the partitioning columns are
+          stripped from the original data files and represented by directory names, so loading data into a partitioned table involves some
+          sort of transformation or preprocessing.
+        </li>
+      </ul>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="partition_sql">
+
+    <title>SQL Statements for Partitioned Tables</title>
+
+    <conbody>
+
+      <p>
+        In terms of Impala SQL syntax, partitioning affects these statements:
+      </p>
+
+      <ul>
+        <li>
+          <codeph><xref href="impala_create_table.xml#create_table">CREATE TABLE</xref></codeph>: you specify a <codeph>PARTITIONED
+          BY</codeph> clause when creating the table to identify names and data types of the partitioning columns. These columns are not
+          included in the main list of columns for the table.
+        </li>
+
+        <li rev="2.5.0">
+          In <keyword keyref="impala25_full"/> and higher, you can also use the <codeph>PARTITIONED BY</codeph> clause in a <codeph>CREATE TABLE AS
+          SELECT</codeph> statement. This syntax lets you use a single statement to create a partitioned table, copy data into it, and
+          create new partitions based on the values in the inserted data.
+        </li>
+
+        <li>
+          <codeph><xref href="impala_alter_table.xml#alter_table">ALTER TABLE</xref></codeph>: you can add or drop partitions, to work with
+          different portions of a huge data set. You can designate the HDFS directory that holds the data files for a specific partition.
+          With data partitioned by date values, you might <q>age out</q> data that is no longer relevant.
+          <note conref="../shared/impala_common.xml#common/add_partition_set_location"/>
+        </li>
+
+        <li>
+          <codeph><xref href="impala_insert.xml#insert">INSERT</xref></codeph>: When you insert data into a partitioned table, you identify
+          the partitioning columns. One or more values from each inserted row are not stored in data files, but instead determine the
+          directory where that row value is stored. You can also specify which partition to load a set of data into, with <codeph>INSERT
+          OVERWRITE</codeph> statements; you can replace the contents of a specific partition but you cannot append data to a specific
+          partition.
+          <p rev="1.3.1" conref="../shared/impala_common.xml#common/insert_inherit_permissions"/>
+        </li>
+
+        <li>
+          Although the syntax of the <codeph><xref href="impala_select.xml#select">SELECT</xref></codeph> statement is the same whether or
+          not the table is partitioned, the way queries interact with partitioned tables can have a dramatic impact on performance and
+          scalability. The mechanism that lets queries skip certain partitions during a query is known as partition pruning; see
+          <xref href="impala_partitioning.xml#partition_pruning"/> for details.
+        </li>
+
+        <li rev="1.4.0">
+          In Impala 1.4 and later, there is a <codeph>SHOW PARTITIONS</codeph> statement that displays information about each partition in a
+          table. See <xref href="impala_show.xml#show"/> for details.
+        </li>
+      </ul>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="partition_static_dynamic">
+
+    <title>Static and Dynamic Partitioning Clauses</title>
+
+    <conbody>
+
+      <p>
+        Specifying all the partition columns in a SQL statement is called <term>static partitioning</term>, because the statement affects a
+        single predictable partition. For example, you use static partitioning with an <codeph>ALTER TABLE</codeph> statement that affects
+        only one partition, or with an <codeph>INSERT</codeph> statement that inserts all values into the same partition:
+      </p>
+
+<codeblock>insert into t1 <b>partition(x=10, y='a')</b> select c1 from some_other_table;
+</codeblock>
+
+      <p>
+        When you specify some partition key columns in an <codeph>INSERT</codeph> statement, but leave out the values, Impala determines
+        which partition to insert. This technique is called <term>dynamic partitioning</term>:
+      </p>
+
+<codeblock>insert into t1 <b>partition(x, y='b')</b> select c1, c2 from some_other_table;
+-- Create new partition if necessary based on variable year, month, and day; insert a single value.
+insert into weather <b>partition (year, month, day)</b> select 'cloudy',2014,4,21;
+-- Create new partition if necessary for specified year and month but variable day; insert a single value.
+insert into weather <b>partition (year=2014, month=04, day)</b> select 'sunny',22;
+</codeblock>
+
+      <p>
+        The more key columns you specify in the <codeph>PARTITION</codeph> clause, the fewer columns you need in the <codeph>SELECT</codeph>
+        list. The trailing columns in the <codeph>SELECT</codeph> list are substituted in order for the partition key columns with no
+        specified value.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="partition_refresh" rev="2.7.0 IMPALA-1683 CDH-43732">
+
+    <title>Refreshing a Single Partition</title>
+
+    <conbody>
+
+      <p>
+        The <codeph>REFRESH</codeph> statement is typically used with partitioned tables when new data files are loaded into a partition by
+        some non-Impala mechanism, such as a Hive or Spark job. The <codeph>REFRESH</codeph> statement makes Impala aware of the new data
+        files so that they can be used in Impala queries. Because partitioned tables typically contain a high volume of data, the
+        <codeph>REFRESH</codeph> operation for a full partitioned table can take significant time.
+      </p>
+
+      <p>
+        In <keyword keyref="impala27_full"/> and higher, you can include a <codeph>PARTITION (<varname>partition_spec</varname>)</codeph> clause in the
+        <codeph>REFRESH</codeph> statement so that only a single partition is refreshed. For example, <codeph>REFRESH big_table PARTITION
+        (year=2017, month=9, day=30)</codeph>. The partition spec must include all the partition key columns. See
+        <xref href="impala_refresh.xml#refresh"/> for more details and examples of <codeph>REFRESH</codeph> syntax and usage.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="partition_permissions">
+
+    <title>Permissions for Partition Subdirectories</title>
+
+    <conbody>
+
+      <p rev="1.3.1" conref="../shared/impala_common.xml#common/insert_inherit_permissions"/>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="partition_pruning">
+
+    <title>Partition Pruning for Queries</title>
+
+    <conbody>
+
+      <p>
+        Partition pruning refers to the mechanism where a query can skip reading the data files corresponding to one or more partitions. If
+        you can arrange for queries to prune large numbers of unnecessary partitions from the query execution plan, the queries use fewer
+        resources and are thus proportionally faster and more scalable.
+      </p>
+
+      <p>
+        For example, if a table is partitioned by columns <codeph>YEAR</codeph>, <codeph>MONTH</codeph>, and <codeph>DAY</codeph>, then
+        <codeph>WHERE</codeph> clauses such as <codeph>WHERE year = 2013</codeph>, <codeph>WHERE year &lt; 2010</codeph>, or <codeph>WHERE
+        year BETWEEN 1995 AND 1998</codeph> allow Impala to skip the data files in all partitions outside the specified range. Likewise,
+        <codeph>WHERE year = 2013 AND month BETWEEN 1 AND 3</codeph> could prune even more partitions, reading the data files for only a
+        portion of one year.
+      </p>
+
+      <p outputclass="toc inpage"/>
+
+    </conbody>
+
+    <concept id="partition_pruning_checking">
+
+      <title>Checking if Partition Pruning Happens for a Query</title>
+
+      <conbody>
+
+        <p>
+          To check the effectiveness of partition pruning for a query, check the <codeph>EXPLAIN</codeph> output for the query before
+          running it. For example, this example shows a table with 3 partitions, where the query only reads 1 of them. The notation
+          <codeph>#partitions=1/3</codeph> in the <codeph>EXPLAIN</codeph> plan confirms that Impala can do the appropriate partition
+          pruning.
+        </p>
+
+<codeblock>[localhost:21000] &gt; insert into census partition (year=2010) values ('Smith'),('Jones');
+[localhost:21000] &gt; insert into census partition (year=2011) values ('Smith'),('Jones'),('Doe');
+[localhost:21000] &gt; insert into census partition (year=2012) values ('Smith'),('Doe');
+[localhost:21000] &gt; select name from census where year=2010;
++-------+
+| name  |
++-------+
+| Smith |
+| Jones |
++-------+
+[localhost:21000] &gt; explain select name from census <b>where year=2010</b>;
++------------------------------------------------------------------+
+| Explain String                                                   |
++------------------------------------------------------------------+
+| PLAN FRAGMENT 0                                                  |
+|   PARTITION: UNPARTITIONED                                       |
+|                                                                  |
+|   1:EXCHANGE                                                     |
+|                                                                  |
+| PLAN FRAGMENT 1                                                  |
+|   PARTITION: RANDOM                                              |
+|                                                                  |
+|   STREAM DATA SINK                                               |
+|     EXCHANGE ID: 1                                               |
+|     UNPARTITIONED                                                |
+|                                                                  |
+|   0:SCAN HDFS                                                    |
+|      table=predicate_propagation.census <b>#partitions=1/3</b> size=12B |
++------------------------------------------------------------------+</codeblock>
+
+        <p rev="1.4.0">
+          For a report of the volume of data that was actually read and processed at each stage of the query, check the output of the
+          <codeph>SUMMARY</codeph> command immediately after running the query. For a more detailed analysis, look at the output of the
+          <codeph>PROFILE</codeph> command; it includes this same summary report near the start of the profile output.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="partition_pruning_sql">
+
+      <title>What SQL Constructs Work with Partition Pruning</title>
+
+      <conbody>
+
+        <p rev="1.2.2">
+          <indexterm audience="Cloudera">predicate propagation</indexterm>
+          Impala can even do partition pruning in cases where the partition key column is not directly compared to a constant, by applying
+          the transitive property to other parts of the <codeph>WHERE</codeph> clause. This technique is known as predicate propagation, and
+          is available in Impala 1.2.2 and later. In this example, the census table includes another column indicating when the data was
+          collected, which happens in 10-year intervals. Even though the query does not compare the partition key column
+          (<codeph>YEAR</codeph>) to a constant value, Impala can deduce that only the partition <codeph>YEAR=2010</codeph> is required, and
+          again only reads 1 out of 3 partitions.
+        </p>
+
+<codeblock rev="1.2.2">[localhost:21000] &gt; drop table census;
+[localhost:21000] &gt; create table census (name string, census_year int) partitioned by (year int);
+[localhost:21000] &gt; insert into census partition (year=2010) values ('Smith',2010),('Jones',2010);
+[localhost:21000] &gt; insert into census partition (year=2011) values ('Smith',2020),('Jones',2020),('Doe',2020);
+[localhost:21000] &gt; insert into census partition (year=2012) values ('Smith',2020),('Doe',2020);
+[localhost:21000] &gt; select name from census where year = census_year and census_year=2010;
++-------+
+| name  |
++-------+
+| Smith |
+| Jones |
++-------+
+[localhost:21000] &gt; explain select name from census <b>where year = census_year and census_year=2010</b>;
++------------------------------------------------------------------+
+| Explain String                                                   |
++------------------------------------------------------------------+
+| PLAN FRAGMENT 0                                                  |
+|   PARTITION: UNPARTITIONED                                       |
+|                                                                  |
+|   1:EXCHANGE                                                     |
+|                                                                  |
+| PLAN FRAGMENT 1                                                  |
+|   PARTITION: RANDOM                                              |
+|                                                                  |
+|   STREAM DATA SINK                                               |
+|     EXCHANGE ID: 1                                               |
+|     UNPARTITIONED                                                |
+|                                                                  |
+|   0:SCAN HDFS                                                    |
+|      table=predicate_propagation.census <b>#partitions=1/3</b> size=22B |
+|      predicates: census_year = 2010, year = census_year          |
++------------------------------------------------------------------+
+</codeblock>
+
+        <p conref="../shared/impala_common.xml#common/partitions_and_views"/>
+
+        <p conref="../shared/impala_common.xml#common/analytic_partition_pruning_caveat"/>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="dynamic_partition_pruning">
+
+      <title>Dynamic Partition Pruning</title>
+
+      <conbody>
+
+        <p>
+          The original mechanism uses to prune partitions is <term>static partition pruning</term>, in which the conditions in the
+          <codeph>WHERE</codeph> clause are analyzed to determine in advance which partitions can be safely skipped. In Impala 2.5 / CDH 5.7
+          and higher, Impala can perform <term>dynamic partition pruning</term>, where information about the partitions is collected during
+          the query, and Impala prunes unnecessary partitions in ways that were impractical to predict in advance.
+        </p>
+
+        <p>
+          For example, if partition key columns are compared to literal values in a <codeph>WHERE</codeph> clause, Impala can perform static
+          partition pruning during the planning phase to only read the relevant partitions:
+        </p>
+
+<codeblock>
+-- The query only needs to read 3 partitions whose key values are known ahead of time.
+-- That's static partition pruning.
+SELECT COUNT(*) FROM sales_table WHERE year IN (2005, 2010, 2015);
+</codeblock>
+
+        <p>
+          Dynamic partition pruning involves using information only available at run time, such as the result of a subquery:
+        </p>
+
+<codeblock conref="../shared/impala_common.xml#common/simple_dpp_example"/>
+
+<!-- Former example. Not sure it really would trigger DPP. SELECT COUNT(*) FROM sales_table WHERE year = (SELECT MAX(year) FROM some_other_table); -->
+
+        <p>
+          In this case, Impala evaluates the subquery, sends the subquery results to all Impala nodes participating in the query, and then
+          each <cmdname>impalad</cmdname> daemon uses the dynamic partition pruning optimization to read only the partitions with the
+          relevant key values.
+        </p>
+
+        <p>
+          Dynamic partition pruning is especially effective for queries involving joins of several large partitioned tables. Evaluating the
+          <codeph>ON</codeph> clauses of the join predicates might normally require reading data from all partitions of certain tables. If
+          the <codeph>WHERE</codeph> clauses of the query refer to the partition key columns, Impala can now often skip reading many of the
+          partitions while evaluating the <codeph>ON</codeph> clauses. The dynamic partition pruning optimization reduces the amount of I/O
+          and the amount of intermediate data stored and transmitted across the network during the query.
+        </p>
+
+        <p conref="../shared/impala_common.xml#common/spill_to_disk_vs_dynamic_partition_pruning"/>
+
+        <p>
+          Dynamic partition pruning is part of the runtime filtering feature, which applies to other kinds of queries in addition to queries
+          against partitioned tables. See <xref href="impala_runtime_filtering.xml#runtime_filtering"/> for full details about this feature.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+  </concept>
+
+  <concept id="partition_key_columns">
+
+    <title>Partition Key Columns</title>
+
+    <conbody>
+
+      <p>
+        The columns you choose as the partition keys should be ones that are frequently used to filter query results in important,
+        large-scale queries. Popular examples are some combination of year, month, and day when the data has associated time values, and
+        geographic region when the data is associated with some place.
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            For time-based data, split out the separate parts into their own columns, because Impala cannot partition based on a
+            <codeph>TIMESTAMP</codeph> column.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            The data type of the partition columns does not have a significant effect on the storage required, because the values from those
+            columns are not stored in the data files, rather they are represented as strings inside HDFS directory names.
+          </p>
+        </li>
+
+        <li rev="IMPALA-2499">
+          <p>
+            In <keyword keyref="impala25_full"/> and higher, you can enable the <codeph>OPTIMIZE_PARTITION_KEY_SCANS</codeph> query option to speed up
+            queries that only refer to partition key columns, such as <codeph>SELECT MAX(year)</codeph>. This setting is not enabled by
+            default because the query behavior is slightly different if the table contains partition directories without actual data inside.
+            See <xref href="impala_optimize_partition_key_scans.xml#optimize_partition_key_scans"/> for details.
+          </p>
+        </li>
+
+        <li>
+          <p conref="../shared/impala_common.xml#common/complex_types_partitioning"/>
+        </li>
+
+        <li>
+          <p>
+            Remember that when Impala queries data stored in HDFS, it is most efficient to use multi-megabyte files to take advantage of the
+            HDFS block size. For Parquet tables, the block size (and ideal size of the data files) is <ph rev="parquet_block_size">256 MB in
+            Impala 2.0 and later</ph>. Therefore, avoid specifying too many partition key columns, which could result in individual
+            partitions containing only small amounts of data. For example, if you receive 1 GB of data per day, you might partition by year,
+            month, and day; while if you receive 5 GB of data per minute, you might partition by year, month, day, hour, and minute. If you
+            have data with a geographic component, you might partition based on postal code if you have many megabytes of data for each
+            postal code, but if not, you might partition by some larger region such as city, state, or country. state
+          </p>
+        </li>
+      </ul>
+
+      <p conref="../shared/impala_common.xml#common/partition_key_optimization"/>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="mixed_format_partitions">
+
+    <title>Setting Different File Formats for Partitions</title>
+
+    <conbody>
+
+      <p>
+        Partitioned tables have the flexibility to use different file formats for different partitions. (For background information about
+        the different file formats Impala supports, see <xref href="impala_file_formats.xml#file_formats"/>.) For example, if you originally
+        received data in text format, then received new data in RCFile format, and eventually began receiving data in Parquet format, all
+        that data could reside in the same table for queries. You just need to ensure that the table is structured so that the data files
+        that use different file formats reside in separate partitions.
+      </p>
+
+      <p>
+        For example, here is how you might switch from text to Parquet data as you receive data for different years:
+      </p>
+
+<codeblock>[localhost:21000] &gt; create table census (name string) partitioned by (year smallint);
+[localhost:21000] &gt; alter table census add partition (year=2012); -- Text format;
+
+[localhost:21000] &gt; alter table census add partition (year=2013); -- Text format switches to Parquet before data loaded;
+[localhost:21000] &gt; alter table census partition (year=2013) set fileformat parquet;
+
+[localhost:21000] &gt; insert into census partition (year=2012) values ('Smith'),('Jones'),('Lee'),('Singh');
+[localhost:21000] &gt; insert into census partition (year=2013) values ('Flores'),('Bogomolov'),('Cooper'),('Appiah');</codeblock>
+
+      <p>
+        At this point, the HDFS directory for <codeph>year=2012</codeph> contains a text-format data file, while the HDFS directory for
+        <codeph>year=2013</codeph> contains a Parquet data file. As always, when loading non-trivial data, you would use <codeph>INSERT ...
+        SELECT</codeph> or <codeph>LOAD DATA</codeph> to import data in large batches, rather than <codeph>INSERT ... VALUES</codeph> which
+        produces small files that are inefficient for real-world queries.
+      </p>
+
+      <p>
+        For other file types that Impala cannot create natively, you can switch into Hive and issue the <codeph>ALTER TABLE ... SET
+        FILEFORMAT</codeph> statements and <codeph>INSERT</codeph> or <codeph>LOAD DATA</codeph> statements there. After switching back to
+        Impala, issue a <codeph>REFRESH <varname>table_name</varname></codeph> statement so that Impala recognizes any partitions or new
+        data added through Hive.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="partition_management">
+
+    <title>Managing Partitions</title>
+
+    <conbody>
+
+      <p>
+        You can add, drop, set the expected file format, or set the HDFS location of the data files for individual partitions within an
+        Impala table. See <xref href="impala_alter_table.xml#alter_table"/> for syntax details, and
+        <xref href="impala_partitioning.xml#mixed_format_partitions"/> for tips on managing tables containing partitions with different file
+        formats.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/add_partition_set_location"/>
+
+      <p>
+        What happens to the data files when a partition is dropped depends on whether the partitioned table is designated as internal or
+        external. For an internal (managed) table, the data files are deleted. For example, if data in the partitioned table is a copy of
+        raw data files stored elsewhere, you might save disk space by dropping older partitions that are no longer required for reporting,
+        knowing that the original data is still available if needed later. For an external table, the data files are left alone. For
+        example, dropping a partition without deleting the associated files lets Impala consider a smaller set of partitions, improving
+        query efficiency and reducing overhead for DDL operations on the table; if the data is needed again later, you can add the partition
+        again. See <xref href="impala_tables.xml#tables" /> for details and examples.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="kudu" id="partition_kudu" audience="Cloudera">
+
+    <title>Using Partitioning with Kudu Tables</title>
+
+    <prolog>
+      <metadata>
+        <data name="Category" value="Kudu"/>
+      </metadata>
+    </prolog>
+
+    <conbody>
+
+      <p>
+        Kudu tables use a more fine-grained partitioning scheme than tables containing HDFS data files. You specify a <codeph>DISTRIBUTE
+        BY</codeph> clause with the <codeph>CREATE TABLE</codeph> statement to identify how to divide the values from the partition key
+        columns.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_perf_benchmarking.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_perf_benchmarking.xml b/docs/topics/impala_perf_benchmarking.xml
new file mode 100644
index 0000000..b2e058d
--- /dev/null
+++ b/docs/topics/impala_perf_benchmarking.xml
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="perf_benchmarks">
+
+  <title>Benchmarking Impala Queries</title>
+  <titlealts audience="PDF"><navtitle>Benchmarking</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Proof of Concept"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Because Impala, like other Hadoop components, is designed to handle large data volumes in a distributed
+      environment, conduct any performance tests using realistic data and cluster configurations. Use a multi-node
+      cluster rather than a single node; run queries against tables containing terabytes of data rather than tens
+      of gigabytes. The parallel processing techniques used by Impala are most appropriate for workloads that are
+      beyond the capacity of a single server.
+    </p>
+
+    <p>
+      When you run queries returning large numbers of rows, the CPU time to pretty-print the output can be
+      substantial, giving an inaccurate measurement of the actual query time. Consider using the
+      <codeph>-B</codeph> option on the <codeph>impala-shell</codeph> command to turn off the pretty-printing, and
+      optionally the <codeph>-o</codeph> option to store query results in a file rather than printing to the
+      screen. See <xref href="impala_shell_options.xml#shell_options"/> for details.
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_perf_cookbook.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_perf_cookbook.xml b/docs/topics/impala_perf_cookbook.xml
new file mode 100644
index 0000000..a42f7c9
--- /dev/null
+++ b/docs/topics/impala_perf_cookbook.xml
@@ -0,0 +1,269 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="perf_cookbook">
+
+  <title>Impala Performance Guidelines and Best Practices</title>
+  <titlealts audience="PDF"><navtitle>Performance Best Practices</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Planning"/>
+      <data name="Category" value="Proof of Concept"/>
+      <data name="Category" value="Guidelines"/>
+      <data name="Category" value="Best Practices"/>
+      <data name="Category" value="Proof of Concept"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Here are performance guidelines and best practices that you can use during planning, experimentation, and
+      performance tuning for an Impala-enabled CDH cluster. All of this information is also available in more
+      detail elsewhere in the Impala documentation; it is gathered together here to serve as a cookbook and
+      emphasize which performance techniques typically provide the highest return on investment
+    </p>
+
+    <p outputclass="toc inpage"/>
+
+    <section id="perf_cookbook_file_format">
+
+      <title>Choose the appropriate file format for the data.</title>
+
+      <p>
+        Typically, for large volumes of data (multiple gigabytes per table or partition), the Parquet file format
+        performs best because of its combination of columnar storage layout, large I/O request size, and
+        compression and encoding. See <xref href="impala_file_formats.xml#file_formats"/> for comparisons of all
+        file formats supported by Impala, and <xref href="impala_parquet.xml#parquet"/> for details about the
+        Parquet file format.
+      </p>
+
+      <note>
+        For smaller volumes of data, a few gigabytes or less for each table or partition, you might not see
+        significant performance differences between file formats. At small data volumes, reduced I/O from an
+        efficient compressed file format can be counterbalanced by reduced opportunity for parallel execution. When
+        planning for a production deployment or conducting benchmarks, always use realistic data volumes to get a
+        true picture of performance and scalability.
+      </note>
+    </section>
+
+    <section id="perf_cookbook_small_files">
+
+      <title>Avoid data ingestion processes that produce many small files.</title>
+
+      <p>
+        When producing data files outside of Impala, prefer either text format or Avro, where you can build up the
+        files row by row. Once the data is in Impala, you can convert it to the more efficient Parquet format and
+        split into multiple data files using a single <codeph>INSERT ... SELECT</codeph> statement. Or, if you have
+        the infrastructure to produce multi-megabyte Parquet files as part of your data preparation process, do
+        that and skip the conversion step inside Impala.
+      </p>
+
+      <p>
+        Always use <codeph>INSERT ... SELECT</codeph> to copy significant volumes of data from table to table
+        within Impala. Avoid <codeph>INSERT ... VALUES</codeph> for any substantial volume of data or
+        performance-critical tables, because each such statement produces a separate tiny data file. See
+        <xref href="impala_insert.xml#insert"/> for examples of the <codeph>INSERT ... SELECT</codeph> syntax.
+      </p>
+
+      <p>
+        For example, if you have thousands of partitions in a Parquet table, each with less than
+        <ph rev="parquet_block_size">256 MB</ph> of data, consider partitioning in a less granular way, such as by
+        year / month rather than year / month / day. If an inefficient data ingestion process produces thousands of
+        data files in the same table or partition, consider compacting the data by performing an <codeph>INSERT ...
+        SELECT</codeph> to copy all the data to a different table; the data will be reorganized into a smaller
+        number of larger files by this process.
+      </p>
+    </section>
+
+    <section id="perf_cookbook_partitioning">
+
+      <title>Choose partitioning granularity based on actual data volume.</title>
+
+      <p>
+        Partitioning is a technique that physically divides the data based on values of one or more columns, such
+        as by year, month, day, region, city, section of a web site, and so on. When you issue queries that request
+        a specific value or range of values for the partition key columns, Impala can avoid reading the irrelevant
+        data, potentially yielding a huge savings in disk I/O.
+      </p>
+
+      <p>
+        When deciding which column(s) to use for partitioning, choose the right level of granularity. For example,
+        should you partition by year, month, and day, or only by year and month? Choose a partitioning strategy
+        that puts at least <ph rev="parquet_block_size">256 MB</ph> of data in each partition, to take advantage of
+        HDFS bulk I/O and Impala distributed queries.
+      </p>
+
+      <p>
+        Over-partitioning can also cause query planning to take longer than necessary, as Impala prunes the
+        unnecessary partitions. Ideally, keep the number of partitions in the table under 30 thousand.
+      </p>
+
+      <p>
+        When preparing data files to go in a partition directory, create several large files rather than many small
+        ones. If you receive data in the form of many small files and have no control over the input format,
+        consider using the <codeph>INSERT ... SELECT</codeph> syntax to copy data from one table or partition to
+        another, which compacts the files into a relatively small number (based on the number of nodes in the
+        cluster).
+      </p>
+
+      <p>
+        If you need to reduce the overall number of partitions and increase the amount of data in each partition,
+        first look for partition key columns that are rarely referenced or are referenced in non-critical queries
+        (not subject to an SLA). For example, your web site log data might be partitioned by year, month, day, and
+        hour, but if most queries roll up the results by day, perhaps you only need to partition by year, month,
+        and day.
+      </p>
+
+      <p>
+        If you need to reduce the granularity even more, consider creating <q>buckets</q>, computed values
+        corresponding to different sets of partition key values. For example, you can use the
+        <codeph>TRUNC()</codeph> function with a <codeph>TIMESTAMP</codeph> column to group date and time values
+        based on intervals such as week or quarter. See
+        <xref href="impala_datetime_functions.xml#datetime_functions"/> for details.
+      </p>
+
+      <p>
+        See <xref href="impala_partitioning.xml#partitioning"/> for full details and performance considerations for
+        partitioning.
+      </p>
+    </section>
+
+    <section id="perf_cookbook_partition_keys">
+
+      <title>Use smallest appropriate integer types for partition key columns.</title>
+
+      <p>
+        Although it is tempting to use strings for partition key columns, since those values are turned into HDFS
+        directory names anyway, you can minimize memory usage by using numeric values for common partition key
+        fields such as <codeph>YEAR</codeph>, <codeph>MONTH</codeph>, and <codeph>DAY</codeph>. Use the smallest
+        integer type that holds the appropriate range of values, typically <codeph>TINYINT</codeph> for
+        <codeph>MONTH</codeph> and <codeph>DAY</codeph>, and <codeph>SMALLINT</codeph> for <codeph>YEAR</codeph>.
+        Use the <codeph>EXTRACT()</codeph> function to pull out individual date and time fields from a
+        <codeph>TIMESTAMP</codeph> value, and <codeph>CAST()</codeph> the return value to the appropriate integer
+        type.
+      </p>
+    </section>
+
+    <section id="perf_cookbook_parquet_block_size">
+
+      <title>Choose an appropriate Parquet block size.</title>
+
+      <p rev="parquet_block_size">
+        By default, the Impala <codeph>INSERT ... SELECT</codeph> statement creates Parquet files with a 256 MB
+        block size. (This default was changed in Impala 2.0. Formerly, the limit was 1 GB, but Impala made
+        conservative estimates about compression, resulting in files that were smaller than 1 GB.)
+      </p>
+
+      <p>
+        Each Parquet file written by Impala is a single block, allowing the whole file to be processed as a unit by a single host.
+        As you copy Parquet files into HDFS or between HDFS filesystems, use <codeph>hdfs dfs -pb</codeph> to preserve the original
+        block size.
+      </p>
+
+      <p>
+        If there is only one or a few data block in your Parquet table, or in a partition that is the only one
+        accessed by a query, then you might experience a slowdown for a different reason: not enough data to take
+        advantage of Impala's parallel distributed queries. Each data block is processed by a single core on one of
+        the DataNodes. In a 100-node cluster of 16-core machines, you could potentially process thousands of data
+        files simultaneously. You want to find a sweet spot between <q>many tiny files</q> and <q>single giant
+        file</q> that balances bulk I/O and parallel processing. You can set the <codeph>PARQUET_FILE_SIZE</codeph>
+        query option before doing an <codeph>INSERT ... SELECT</codeph> statement to reduce the size of each
+        generated Parquet file. <ph rev="2.0.0">(Specify the file size as an absolute number of bytes, or in Impala
+        2.0 and later, in units ending with <codeph>m</codeph> for megabytes or <codeph>g</codeph> for
+        gigabytes.)</ph> Run benchmarks with different file sizes to find the right balance point for your
+        particular data volume.
+      </p>
+    </section>
+
+    <section id="perf_cookbook_stats">
+
+      <title>Gather statistics for all tables used in performance-critical or high-volume join queries.</title>
+
+      <p>
+        Gather the statistics with the <codeph>COMPUTE STATS</codeph> statement. See
+        <xref href="impala_perf_joins.xml#perf_joins"/> for details.
+      </p>
+    </section>
+
+    <section id="perf_cookbook_network">
+
+      <title>Minimize the overhead of transmitting results back to the client.</title>
+
+      <p>
+        Use techniques such as:
+      </p>
+
+      <ul>
+        <li>
+          Aggregation. If you need to know how many rows match a condition, the total values of matching values
+          from some column, the lowest or highest matching value, and so on, call aggregate functions such as
+          <codeph>COUNT()</codeph>, <codeph>SUM()</codeph>, and <codeph>MAX()</codeph> in the query rather than
+          sending the result set to an application and doing those computations there. Remember that the size of an
+          unaggregated result set could be huge, requiring substantial time to transmit across the network.
+        </li>
+
+        <li>
+          Filtering. Use all applicable tests in the <codeph>WHERE</codeph> clause of a query to eliminate rows
+          that are not relevant, rather than producing a big result set and filtering it using application logic.
+        </li>
+
+        <li>
+          <codeph>LIMIT</codeph> clause. If you only need to see a few sample values from a result set, or the top
+          or bottom values from a query using <codeph>ORDER BY</codeph>, include the <codeph>LIMIT</codeph> clause
+          to reduce the size of the result set rather than asking for the full result set and then throwing most of
+          the rows away.
+        </li>
+
+        <li>
+          Avoid overhead from pretty-printing the result set and displaying it on the screen. When you retrieve the
+          results through <cmdname>impala-shell</cmdname>, use <cmdname>impala-shell</cmdname> options such as
+          <codeph>-B</codeph> and <codeph>--output_delimiter</codeph> to produce results without special
+          formatting, and redirect output to a file rather than printing to the screen. Consider using
+          <codeph>INSERT ... SELECT</codeph> to write the results directly to new files in HDFS. See
+          <xref href="impala_shell_options.xml#shell_options"/> for details about the
+          <cmdname>impala-shell</cmdname> command-line options.
+        </li>
+      </ul>
+    </section>
+
+    <section id="perf_cookbook_explain">
+
+      <title>Verify that your queries are planned in an efficient logical manner.</title>
+
+      <p>
+        Examine the <codeph>EXPLAIN</codeph> plan for a query before actually running it. See
+        <xref href="impala_explain.xml#explain"/> and <xref href="impala_explain_plan.xml#perf_explain"/> for
+        details.
+      </p>
+    </section>
+
+    <section id="perf_cookbook_profile">
+
+      <title>Verify performance characteristics of queries.</title>
+
+      <p>
+        Verify that the low-level aspects of I/O, memory usage, network bandwidth, CPU utilization, and so on are
+        within expected ranges by examining the query profile for a query after running it. See
+        <xref href="impala_explain_plan.xml#perf_profile"/> for details.
+      </p>
+    </section>
+
+    <section id="perf_cookbook_os">
+
+      <title>Use appropriate operating system settings.</title>
+
+      <p>
+        See <xref href="http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cdh_admin_performance.html" scope="external" format="html">Optimizing Performance in CDH</xref>
+        for recommendations about operating system
+        settings that you can change to influence Impala performance. In particular, you might find
+        that changing the <codeph>vm.swappiness</codeph> Linux kernel setting to a non-zero value improves
+        overall performance.
+      </p>
+    </section>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_perf_ddl.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_perf_ddl.xml b/docs/topics/impala_perf_ddl.xml
new file mode 100644
index 0000000..d075cd2
--- /dev/null
+++ b/docs/topics/impala_perf_ddl.xml
@@ -0,0 +1,42 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="perf_ddl">
+
+  <title>Performance Considerations for DDL Statements</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="DDL"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      These tips and guidelines apply to the Impala DDL statements, which are listed in
+      <xref href="impala_ddl.xml#ddl"/>.
+    </p>
+
+    <p>
+      Because Impala DDL statements operate on the metastore database, the performance considerations for those
+      statements are totally different than for distributed queries that operate on HDFS
+      <ph rev="2.2.0">or S3</ph> data files, or on HBase tables.
+    </p>
+
+    <p>
+      Each DDL statement makes a relatively small update to the metastore database. The overhead for each statement
+      is proportional to the overall number of Impala and Hive tables, and (for a partitioned table) to the overall
+      number of partitions in that table. Issuing large numbers of DDL statements (such as one for each table or
+      one for each partition) also has the potential to encounter a bottleneck with access to the metastore
+      database. Therefore, for efficient DDL, try to design your application logic and ETL pipeline to avoid a huge
+      number of tables and a huge number of partitions within each table. In this context, <q>huge</q> is in the
+      range of tens of thousands or hundreds of thousands.
+    </p>
+
+    <note conref="../shared/impala_common.xml#common/add_partition_set_location"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_perf_hdfs_caching.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_perf_hdfs_caching.xml b/docs/topics/impala_perf_hdfs_caching.xml
new file mode 100644
index 0000000..883bc8d
--- /dev/null
+++ b/docs/topics/impala_perf_hdfs_caching.xml
@@ -0,0 +1,609 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.4" id="hdfs_caching">
+
+  <title>Using HDFS Caching with Impala (<keyword keyref="impala21"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>HDFS Caching</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Scalability"/>
+      <data name="Category" value="HDFS"/>
+      <data name="Category" value="HDFS Caching"/>
+      <data name="Category" value="Memory"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      HDFS caching provides performance and scalability benefits in production environments where Impala queries
+      and other Hadoop jobs operate on quantities of data much larger than the physical RAM on the DataNodes,
+      making it impractical to rely on the Linux OS cache, which only keeps the most recently used data in memory.
+      Data read from the HDFS cache avoids the overhead of checksumming and memory-to-memory copying involved when
+      using data from the Linux OS cache.
+    </p>
+
+    <note>
+      <p>
+        On a small or lightly loaded cluster, HDFS caching might not produce any speedup. It might even lead to
+        slower queries, if I/O read operations that were performed in parallel across the entire cluster are replaced by in-memory
+        operations operating on a smaller number of hosts. The hosts where the HDFS blocks are cached can become
+        bottlenecks because they experience high CPU load while processing the cached data blocks, while other hosts remain idle.
+        Therefore, always compare performance with and without this feature enabled, using a realistic workload.
+      </p>
+      <p rev="2.2.0">
+        In <keyword keyref="impala22_full"/> and higher, you can spread the CPU load more evenly by specifying the <codeph>WITH REPLICATION</codeph>
+        clause of the <codeph>CREATE TABLE</codeph> and <codeph>ALTER TABLE</codeph> statements.
+        This clause lets you control the replication factor for
+        HDFS caching for a specific table or partition. By default, each cached block is
+        only present on a single host, which can lead to CPU contention if the same host
+        processes each cached block. Increasing the replication factor lets Impala choose
+        different hosts to process different cached blocks, to better distribute the CPU load.
+        Always use a <codeph>WITH REPLICATION</codeph> setting of at least 3, and adjust upward
+        if necessary to match the replication factor for the underlying HDFS data files.
+      </p>
+      <p rev="2.5.0">
+        In <keyword keyref="impala25_full"/> and higher, Impala automatically randomizes which host processes
+        a cached HDFS block, to avoid CPU hotspots. For tables where HDFS caching is not applied,
+        Impala designates which host to process a data block using an algorithm that estimates
+        the load on each host. If CPU hotspots still arise during queries,
+        you can enable additional randomization for the scheduling algorithm for non-HDFS cached data
+        by setting the <codeph>SCHEDULE_RANDOM_REPLICA</codeph> query option.
+      </p>
+    </note>
+
+    <p outputclass="toc inpage"/>
+
+<!-- Could conref this background link; haven't decided yet the best place or if it's needed twice. -->
+
+    <p>
+      For background information about how to set up and manage HDFS caching for a CDH cluster, see
+<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh_ig_hdfs_caching.html -->
+      <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_hdfs_caching.html" scope="external" format="html">the
+      CDH documentation</xref>.
+    </p>
+  </conbody>
+
+  <concept id="hdfs_caching_overview">
+
+    <title>Overview of HDFS Caching for Impala</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Concepts"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        On <ph rev="upstream">CDH 5.1</ph> and higher, Impala can use the HDFS caching feature to make more effective use of RAM, so that
+        repeated queries can take advantage of data <q>pinned</q> in memory regardless of how much data is
+        processed overall. The HDFS caching feature lets you designate a subset of frequently accessed data to be
+        pinned permanently in memory, remaining in the cache across multiple queries and never being evicted. This
+        technique is suitable for tables or partitions that are frequently accessed and are small enough to fit
+        entirely within the HDFS memory cache. For example, you might designate several dimension tables to be
+        pinned in the cache, to speed up many different join queries that reference them. Or in a partitioned
+        table, you might pin a partition holding data from the most recent time period because that data will be
+        queried intensively; then when the next set of data arrives, you could unpin the previous partition and pin
+        the partition holding the new data.
+      </p>
+
+      <p>
+        Because this Impala performance feature relies on HDFS infrastructure, it only applies to Impala tables
+        that use HDFS data files. HDFS caching for Impala does not apply to HBase tables, S3 tables,
+        Kudu tables,
+        or Isilon tables.
+      </p>
+
+    </conbody>
+  </concept>
+
+  <concept id="hdfs_caching_prereqs">
+
+    <title>Setting Up HDFS Caching for Impala</title>
+
+    <conbody>
+
+      <p>
+        To use HDFS caching with Impala, first set up that feature for your CDH cluster:
+      </p>
+
+      <ul>
+        <li>
+          <p>
+          Decide how much memory to devote to the HDFS cache on each host. Remember that the total memory available
+          for cached data is the sum of the cache sizes on all the hosts. By default, any data block is only cached on one
+          host, although you can cache a block across multiple hosts by increasing the replication factor.
+          <!-- Obsoleted in Impala 2.2 and higher by IMPALA-1587.
+          Once a data block is cached on one host, all requests to process that block are routed to that same host.)
+          -->
+          </p>
+        </li>
+
+        <li>
+          <p>
+          Issue <cmdname>hdfs cacheadmin</cmdname> commands to set up one or more cache pools, owned by the same
+          user as the <cmdname>impalad</cmdname> daemon (typically <codeph>impala</codeph>). For example:
+<codeblock>hdfs cacheadmin -addPool four_gig_pool -owner impala -limit 4000000000
+</codeblock>
+          For details about the <cmdname>hdfs cacheadmin</cmdname> command, see
+<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh_ig_hdfs_caching.html -->
+          <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_hdfs_caching.html" scope="external" format="html">the
+          CDH documentation</xref>.
+          </p>
+        </li>
+      </ul>
+
+      <p>
+        Once HDFS caching is enabled and one or more pools are available, see
+        <xref href="impala_perf_hdfs_caching.xml#hdfs_caching_ddl"/> for how to choose which Impala data to load
+        into the HDFS cache. On the Impala side, you specify the cache pool name defined by the <codeph>hdfs
+        cacheadmin</codeph> command in the Impala DDL statements that enable HDFS caching for a table or partition,
+        such as <codeph>CREATE TABLE ... CACHED IN <varname>pool</varname></codeph> or <codeph>ALTER TABLE ... SET
+        CACHED IN <varname>pool</varname></codeph>.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="hdfs_caching_ddl">
+
+    <title>Enabling HDFS Caching for Impala Tables and Partitions</title>
+
+    <conbody>
+
+      <p>
+        Begin by choosing which tables or partitions to cache. For example, these might be lookup tables that are
+        accessed by many different join queries, or partitions corresponding to the most recent time period that
+        are analyzed by different reports or ad hoc queries.
+      </p>
+
+      <p>
+        In your SQL statements, you specify logical divisions such as tables and partitions to be cached. Impala
+        translates these requests into HDFS-level directives that apply to particular directories and files. For
+        example, given a partitioned table <codeph>CENSUS</codeph> with a partition key column
+        <codeph>YEAR</codeph>, you could choose to cache all or part of the data as follows:
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/impala_cache_replication_factor"/>
+
+<codeblock>-- Cache the entire table (all partitions).
+alter table census set cached in '<varname>pool_name</varname>';
+
+-- Remove the entire table from the cache.
+alter table census set uncached;
+
+-- Cache a portion of the table (a single partition).
+-- If the table is partitioned by multiple columns (such as year, month, day),
+-- the ALTER TABLE command must specify values for all those columns.
+alter table census partition (year=1960) set cached in '<varname>pool_name</varname>';
+
+<ph rev="2.2.0">-- Cache the data from one partition on up to 4 hosts, to minimize CPU load on any
+-- single host when the same data block is processed multiple times.
+alter table census partition (year=1970)
+  set cached in '<varname>pool_name</varname>' with replication = 4;</ph>
+
+-- At each stage, check the volume of cached data.
+-- For large tables or partitions, the background loading might take some time,
+-- so you might have to wait and reissue the statement until all the data
+-- has finished being loaded into the cache.
+show table stats census;
++-------+-------+--------+------+--------------+--------+
+| year  | #Rows | #Files | Size | Bytes Cached | Format |
++-------+-------+--------+------+--------------+--------+
+| 1900  | -1    | 1      | 11B  | NOT CACHED   | TEXT   |
+| 1940  | -1    | 1      | 11B  | NOT CACHED   | TEXT   |
+| 1960  | -1    | 1      | 11B  | 11B          | TEXT   |
+| 1970  | -1    | 1      | 11B  | NOT CACHED   | TEXT   |
+| Total | -1    | 4      | 44B  | 11B          |        |
++-------+-------+--------+------+--------------+--------+
+</codeblock>
+
+      <p>
+        <b>CREATE TABLE considerations:</b>
+      </p>
+
+      <p>
+        The HDFS caching feature affects the Impala <codeph>CREATE TABLE</codeph> statement as follows:
+      </p>
+
+      <ul>
+        <li>
+        <p>
+          You can put a <codeph>CACHED IN '<varname>pool_name</varname>'</codeph> clause
+          <ph rev="2.2.0">and optionally a <codeph>WITH REPLICATION = <varname>number_of_hosts</varname></codeph> clause</ph>
+          at the end of a
+          <codeph>CREATE TABLE</codeph> statement to automatically cache the entire contents of the table,
+          including any partitions added later. The <varname>pool_name</varname> is a pool that you previously set
+          up with the <cmdname>hdfs cacheadmin</cmdname> command.
+        </p>
+        </li>
+
+        <li>
+        <p>
+          Once a table is designated for HDFS caching through the <codeph>CREATE TABLE</codeph> statement, if new
+          partitions are added later through <codeph>ALTER TABLE ... ADD PARTITION</codeph> statements, the data in
+          those new partitions is automatically cached in the same pool.
+        </p>
+        </li>
+
+        <li>
+        <p>
+          If you want to perform repetitive queries on a subset of data from a large table, and it is not practical
+          to designate the entire table or specific partitions for HDFS caching, you can create a new cached table
+          with just a subset of the data by using <codeph>CREATE TABLE ... CACHED IN '<varname>pool_name</varname>'
+          AS SELECT ... WHERE ...</codeph>. When you are finished with generating reports from this subset of data,
+          drop the table and both the data files and the data cached in RAM are automatically deleted.
+        </p>
+        </li>
+      </ul>
+
+      <p>
+        See <xref href="impala_create_table.xml#create_table"/> for the full syntax.
+      </p>
+
+      <p>
+        <b>Other memory considerations:</b>
+      </p>
+
+      <p>
+        Certain DDL operations, such as <codeph>ALTER TABLE ... SET LOCATION</codeph>, are blocked while the
+        underlying HDFS directories contain cached files. You must uncache the files first, before changing the
+        location, dropping the table, and so on.
+      </p>
+
+      <p>
+        When data is requested to be pinned in memory, that process happens in the background without blocking
+        access to the data while the caching is in progress. Loading the data from disk could take some time.
+        Impala reads each HDFS data block from memory if it has been pinned already, or from disk if it has not
+        been pinned yet. When files are added to a table or partition whose contents are cached, Impala
+        automatically detects those changes and performs a <codeph>REFRESH</codeph> automatically once the relevant
+        data is cached.
+      </p>
+
+      <p>
+        The amount of data that you can pin on each node through the HDFS caching mechanism is subject to a quota
+        that is enforced by the underlying HDFS service. Before requesting to pin an Impala table or partition in
+        memory, check that its size does not exceed this quota.
+      </p>
+
+      <note>
+        Because the HDFS cache consists of combined memory from all the DataNodes in the cluster, cached tables or
+        partitions can be bigger than the amount of HDFS cache memory on any single host.
+      </note>
+    </conbody>
+  </concept>
+
+  <concept id="hdfs_caching_etl">
+
+    <title>Loading and Removing Data with HDFS Caching Enabled</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="ETL"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        When HDFS caching is enabled, extra processing happens in the background when you add or remove data
+        through statements such as <codeph>INSERT</codeph> and <codeph>DROP TABLE</codeph>.
+      </p>
+
+      <p>
+        <b>Inserting or loading data:</b>
+      </p>
+
+      <ul>
+        <li>
+          When Impala performs an <codeph><xref href="impala_insert.xml#insert">INSERT</xref></codeph> or
+          <codeph><xref href="impala_load_data.xml#load_data">LOAD DATA</xref></codeph> statement for a table or
+          partition that is cached, the new data files are automatically cached and Impala recognizes that fact
+          automatically.
+        </li>
+
+        <li>
+          If you perform an <codeph>INSERT</codeph> or <codeph>LOAD DATA</codeph> through Hive, as always, Impala
+          only recognizes the new data files after a <codeph>REFRESH <varname>table_name</varname></codeph>
+          statement in Impala.
+        </li>
+
+        <li>
+          If the cache pool is entirely full, or becomes full before all the requested data can be cached, the
+          Impala DDL statement returns an error. This is to avoid situations where only some of the requested data
+          could be cached.
+        </li>
+
+        <li>
+          When HDFS caching is enabled for a table or partition, new data files are cached automatically when they
+          are added to the appropriate directory in HDFS, without the need for a <codeph>REFRESH</codeph> statement
+          in Impala. Impala automatically performs a <codeph>REFRESH</codeph> once the new data is loaded into the
+          HDFS cache.
+        </li>
+      </ul>
+
+      <p>
+        <b>Dropping tables, partitions, or cache pools:</b>
+      </p>
+
+      <p>
+        The HDFS caching feature interacts with the Impala
+        <codeph><xref href="impala_drop_table.xml#drop_table">DROP TABLE</xref></codeph> and
+        <codeph><xref href="impala_alter_table.xml#alter_table">ALTER TABLE ... DROP PARTITION</xref></codeph>
+        statements as follows:
+      </p>
+
+      <ul>
+        <li>
+          When you issue a <codeph>DROP TABLE</codeph> for a table that is entirely cached, or has some partitions
+          cached, the <codeph>DROP TABLE</codeph> succeeds and all the cache directives Impala submitted for that
+          table are removed from the HDFS cache system.
+        </li>
+
+        <li>
+          The same applies to <codeph>ALTER TABLE ... DROP PARTITION</codeph>. The operation succeeds and any cache
+          directives are removed.
+        </li>
+
+        <li>
+          As always, the underlying data files are removed if the dropped table is an internal table, or the
+          dropped partition is in its default location underneath an internal table. The data files are left alone
+          if the dropped table is an external table, or if the dropped partition is in a non-default location.
+        </li>
+
+        <li>
+          If you designated the data files as cached through the <cmdname>hdfs cacheadmin</cmdname> command, and
+          the data files are left behind as described in the previous item, the data files remain cached. Impala
+          only removes the cache directives submitted by Impala through the <codeph>CREATE TABLE</codeph> or
+          <codeph>ALTER TABLE</codeph> statements. It is OK to have multiple redundant cache directives pertaining
+          to the same files; the directives all have unique IDs and owners so that the system can tell them apart.
+        </li>
+
+        <li>
+          If you drop an HDFS cache pool through the <cmdname>hdfs cacheadmin</cmdname> command, all the Impala
+          data files are preserved, just no longer cached. After a subsequent <codeph>REFRESH</codeph>,
+          <codeph>SHOW TABLE STATS</codeph> reports 0 bytes cached for each associated Impala table or partition.
+        </li>
+      </ul>
+
+      <p>
+        <b>Relocating a table or partition:</b>
+      </p>
+
+      <p>
+        The HDFS caching feature interacts with the Impala
+        <codeph><xref href="impala_alter_table.xml#alter_table">ALTER TABLE ... SET LOCATION</xref></codeph>
+        statement as follows:
+      </p>
+
+      <ul>
+        <li>
+          If you have designated a table or partition as cached through the <codeph>CREATE TABLE</codeph> or
+          <codeph>ALTER TABLE</codeph> statements, subsequent attempts to relocate the table or partition through
+          an <codeph>ALTER TABLE ... SET LOCATION</codeph> statement will fail. You must issue an <codeph>ALTER
+          TABLE ... SET UNCACHED</codeph> statement for the table or partition first. Otherwise, Impala would lose
+          track of some cached data files and have no way to uncache them later.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="hdfs_caching_admin">
+
+    <title>Administration for HDFS Caching with Impala</title>
+
+    <conbody>
+
+      <p>
+        Here are the guidelines and steps to check or change the status of HDFS caching for Impala data:
+      </p>
+
+      <p>
+        <b>hdfs cacheadmin command:</b>
+      </p>
+
+      <ul>
+        <li>
+          If you drop a cache pool with the <cmdname>hdfs cacheadmin</cmdname> command, Impala queries against the
+          associated data files will still work, by falling back to reading the files from disk. After performing a
+          <codeph>REFRESH</codeph> on the table, Impala reports the number of bytes cached as 0 for all associated
+          tables and partitions.
+        </li>
+
+        <li>
+          You might use <cmdname>hdfs cacheadmin</cmdname> to get a list of existing cache pools, or detailed
+          information about the pools, as follows:
+<codeblock scale="60">hdfs cacheadmin -listDirectives         # Basic info
+Found 122 entries
+  ID POOL       REPL EXPIRY  PATH
+ 123 testPool      1 never   /user/hive/warehouse/tpcds.store_sales
+ 124 testPool      1 never   /user/hive/warehouse/tpcds.store_sales/ss_date=1998-01-15
+ 125 testPool      1 never   /user/hive/warehouse/tpcds.store_sales/ss_date=1998-02-01
+...
+
+hdfs cacheadmin -listDirectives -stats  # More details
+Found 122 entries
+  ID POOL       REPL EXPIRY  PATH                                                        BYTES_NEEDED  BYTES_CACHED  FILES_NEEDED  FILES_CACHED
+ 123 testPool      1 never   /user/hive/warehouse/tpcds.store_sales                                 0             0             0             0
+ 124 testPool      1 never   /user/hive/warehouse/tpcds.store_sales/ss_date=1998-01-15         143169        143169             1             1
+ 125 testPool      1 never   /user/hive/warehouse/tpcds.store_sales/ss_date=1998-02-01         112447        112447             1             1
+...
+</codeblock>
+        </li>
+      </ul>
+
+      <p>
+        <b>Impala SHOW statement:</b>
+      </p>
+
+      <ul>
+        <li>
+          For each table or partition, the <codeph>SHOW TABLE STATS</codeph> or <codeph>SHOW PARTITIONS</codeph>
+          statement displays the number of bytes currently cached by the HDFS caching feature. If there are no
+          cache directives in place for that table or partition, the result set displays <codeph>NOT
+          CACHED</codeph>. A value of 0, or a smaller number than the overall size of the table or partition,
+          indicates that the cache request has been submitted but the data has not been entirely loaded into memory
+          yet. See <xref href="impala_show.xml#show"/> for details.
+        </li>
+      </ul>
+
+      <p>
+        <b>Cloudera Manager:</b>
+      </p>
+
+      <ul>
+        <li>
+          You can enable or disable HDFS caching through Cloudera Manager, using the configuration setting
+          <uicontrol>Maximum Memory Used for Caching</uicontrol> for the HDFS service. This control sets the HDFS
+          configuration parameter <codeph>dfs_datanode_max_locked_memory</codeph>, which specifies the upper limit
+          of HDFS cache size on each node.
+        </li>
+
+        <li>
+          All the other manipulation of the HDFS caching settings, such as what files are cached, is done through
+          the command line, either Impala DDL statements or the Linux <cmdname>hdfs cacheadmin</cmdname> command.
+        </li>
+      </ul>
+
+      <p>
+        <b>Impala memory limits:</b>
+      </p>
+
+      <p>
+        The Impala HDFS caching feature interacts with the Impala memory limits as follows:
+      </p>
+
+      <ul>
+        <li>
+          The maximum size of each HDFS cache pool is specified externally to Impala, through the <cmdname>hdfs
+          cacheadmin</cmdname> command.
+        </li>
+
+        <li>
+          All the memory used for HDFS caching is separate from the <cmdname>impalad</cmdname> daemon address space
+          and does not count towards the limits of the <codeph>--mem_limit</codeph> startup option,
+          <codeph>MEM_LIMIT</codeph> query option, or further limits imposed through YARN resource management or
+          the Linux <codeph>cgroups</codeph> mechanism.
+        </li>
+
+        <li>
+          Because accessing HDFS cached data avoids a memory-to-memory copy operation, queries involving cached
+          data require less memory on the Impala side than the equivalent queries on uncached data. In addition to
+          any performance benefits in a single-user environment, the reduced memory helps to improve scalability
+          under high-concurrency workloads.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="hdfs_caching_performance">
+
+    <title>Performance Considerations for HDFS Caching with Impala</title>
+
+    <conbody>
+
+      <p>
+        In Impala 1.4.0 and higher, Impala supports efficient reads from data that is pinned in memory through HDFS
+        caching. Impala takes advantage of the HDFS API and reads the data from memory rather than from disk
+        whether the data files are pinned using Impala DDL statements, or using the command-line mechanism where
+        you specify HDFS paths.
+      </p>
+
+      <p>
+        When you examine the output of the <cmdname>impala-shell</cmdname> <cmdname>SUMMARY</cmdname> command, or
+        look in the metrics report for the <cmdname>impalad</cmdname> daemon, you see how many bytes are read from
+        the HDFS cache. For example, this excerpt from a query profile illustrates that all the data read during a
+        particular phase of the query came from the HDFS cache, because the <codeph>BytesRead</codeph> and
+        <codeph>BytesReadDataNodeCache</codeph> values are identical.
+      </p>
+
+<codeblock>HDFS_SCAN_NODE (id=0):(Total: 11s114ms, non-child: 11s114ms, % non-child: 100.00%)
+        - AverageHdfsReadThreadConcurrency: 0.00
+        - AverageScannerThreadConcurrency: 32.75
+<b>        - BytesRead: 10.47 GB (11240756479)
+        - BytesReadDataNodeCache: 10.47 GB (11240756479)</b>
+        - BytesReadLocal: 10.47 GB (11240756479)
+        - BytesReadShortCircuit: 10.47 GB (11240756479)
+        - DecompressionTime: 27s572ms
+</codeblock>
+
+      <p>
+        For queries involving smaller amounts of data, or in single-user workloads, you might not notice a
+        significant difference in query response time with or without HDFS caching. Even with HDFS caching turned
+        off, the data for the query might still be in the Linux OS buffer cache. The benefits become clearer as
+        data volume increases, and especially as the system processes more concurrent queries. HDFS caching
+        improves the scalability of the overall system. That is, it prevents query performance from declining when
+        the workload outstrips the capacity of the Linux OS cache.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/hdfs_caching_encryption_caveat"/>
+
+      <p>
+        <b>SELECT considerations:</b>
+      </p>
+
+      <p>
+        The Impala HDFS caching feature interacts with the
+        <codeph><xref href="impala_select.xml#select">SELECT</xref></codeph> statement and query performance as
+        follows:
+      </p>
+
+      <ul>
+        <li>
+          Impala automatically reads from memory any data that has been designated as cached and actually loaded
+          into the HDFS cache. (It could take some time after the initial request to fully populate the cache for a
+          table with large size or many partitions.) The speedup comes from two aspects: reading from RAM instead
+          of disk, and accessing the data straight from the cache area instead of copying from one RAM area to
+          another. This second aspect yields further performance improvement over the standard OS caching
+          mechanism, which still results in memory-to-memory copying of cached data.
+        </li>
+
+        <li>
+          For small amounts of data, the query speedup might not be noticeable in terms of wall clock time. The
+          performance might be roughly the same with HDFS caching turned on or off, due to recently used data being
+          held in the Linux OS cache. The difference is more pronounced with:
+          <ul>
+            <li>
+              Data volumes (for all queries running concurrently) that exceed the size of the Linux OS cache.
+            </li>
+
+            <li>
+              A busy cluster running many concurrent queries, where the reduction in memory-to-memory copying and
+              overall memory usage during queries results in greater scalability and throughput.
+            </li>
+
+            <li>
+              Thus, to really exercise and benchmark this feature in a development environment, you might need to
+              simulate realistic workloads and concurrent queries that match your production environment.
+            </li>
+
+            <li>
+              One way to simulate a heavy workload on a lightly loaded system is to flush the OS buffer cache (on
+              each DataNode) between iterations of queries against the same tables or partitions:
+<codeblock>$ sync
+$ echo 1 &gt; /proc/sys/vm/drop_caches
+</codeblock>
+            </li>
+          </ul>
+        </li>
+
+        <li>
+          Impala queries take advantage of HDFS cached data regardless of whether the cache directive was issued by
+          Impala or externally through the <cmdname>hdfs cacheadmin</cmdname> command, for example for an external
+          table where the cached data files might be accessed by several different Hadoop components.
+        </li>
+
+        <li>
+          If your query returns a large result set, the time reported for the query could be dominated by the time
+          needed to print the results on the screen. To measure the time for the underlying query processing, query
+          the <codeph>COUNT()</codeph> of the big result set, which does all the same processing but only prints a
+          single line to the screen.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+</concept>


[14/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_perf_joins.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_perf_joins.xml b/docs/topics/impala_perf_joins.xml
new file mode 100644
index 0000000..4fcd896
--- /dev/null
+++ b/docs/topics/impala_perf_joins.xml
@@ -0,0 +1,506 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="perf_joins">
+
+  <title>Performance Considerations for Join Queries</title>
+  <titlealts audience="PDF"><navtitle>Join Performance</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Queries involving join operations often require more tuning than queries that refer to only one table. The
+      maximum size of the result set from a join query is the product of the number of rows in all the joined
+      tables. When joining several tables with millions or billions of rows, any missed opportunity to filter the
+      result set, or other inefficiency in the query, could lead to an operation that does not finish in a
+      practical time and has to be cancelled.
+    </p>
+
+    <p rev="1.2.2">
+      The simplest technique for tuning an Impala join query is to collect statistics on each table involved in the
+      join using the <codeph><xref href="impala_compute_stats.xml#compute_stats">COMPUTE STATS</xref></codeph>
+      statement, and then let Impala automatically optimize the query based on the size of each table, number of
+      distinct values of each column, and so on. The <codeph>COMPUTE STATS</codeph> statement and the join
+      optimization are new features introduced in Impala 1.2.2. For accurate statistics about each table, issue the
+      <codeph>COMPUTE STATS</codeph> statement after loading the data into that table, and again if the amount of
+      data changes substantially due to an <codeph>INSERT</codeph>, <codeph>LOAD DATA</codeph>, adding a partition,
+      and so on.
+    </p>
+
+    <p>
+      If statistics are not available for all the tables in the join query, or if Impala chooses a join order that
+      is not the most efficient, you can override the automatic join order optimization by specifying the
+      <codeph>STRAIGHT_JOIN</codeph> keyword immediately after the <codeph>SELECT</codeph> keyword. In this case,
+      Impala uses the order the tables appear in the query to guide how the joins are processed.
+    </p>
+
+    <p>
+      When you use the <codeph>STRAIGHT_JOIN</codeph> technique, you must order the tables in the join query
+      manually instead of relying on the Impala optimizer. The optimizer uses sophisticated techniques to estimate
+      the size of the result set at each stage of the join. For manual ordering, use this heuristic approach to
+      start with, and then experiment to fine-tune the order:
+    </p>
+
+    <ul>
+      <li>
+        Specify the largest table first. This table is read from disk by each Impala node and so its size is not
+        significant in terms of memory usage during the query.
+      </li>
+
+      <li>
+        Next, specify the smallest table. The contents of the second, third, and so on tables are all transmitted
+        across the network. You want to minimize the size of the result set from each subsequent stage of the join
+        query. The most likely approach involves joining a small table first, so that the result set remains small
+        even as subsequent larger tables are processed.
+      </li>
+
+      <li>
+        Join the next smallest table, then the next smallest, and so on.
+      </li>
+
+      <li>
+        For example, if you had tables <codeph>BIG</codeph>, <codeph>MEDIUM</codeph>, <codeph>SMALL</codeph>, and
+        <codeph>TINY</codeph>, the logical join order to try would be <codeph>BIG</codeph>, <codeph>TINY</codeph>,
+        <codeph>SMALL</codeph>, <codeph>MEDIUM</codeph>.
+      </li>
+    </ul>
+
+    <p>
+      The terms <q>largest</q> and <q>smallest</q> refers to the size of the intermediate result set based on the
+      number of rows and columns from each table that are part of the result set. For example, if you join one
+      table <codeph>sales</codeph> with another table <codeph>customers</codeph>, a query might find results from
+      100 different customers who made a total of 5000 purchases. In that case, you would specify <codeph>SELECT
+      ... FROM sales JOIN customers ...</codeph>, putting <codeph>customers</codeph> on the right side because it
+      is smaller in the context of this query.
+    </p>
+
+    <p>
+      The Impala query planner chooses between different techniques for performing join queries, depending on the
+      absolute and relative sizes of the tables. <b>Broadcast joins</b> are the default, where the right-hand table
+      is considered to be smaller than the left-hand table, and its contents are sent to all the other nodes
+      involved in the query. The alternative technique is known as a <b>partitioned join</b> (not related to a
+      partitioned table), which is more suitable for large tables of roughly equal size. With this technique,
+      portions of each table are sent to appropriate other nodes where those subsets of rows can be processed in
+      parallel. The choice of broadcast or partitioned join also depends on statistics being available for all
+      tables in the join, gathered by the <codeph>COMPUTE STATS</codeph> statement.
+    </p>
+
+    <p>
+      To see which join strategy is used for a particular query, issue an <codeph>EXPLAIN</codeph> statement for
+      the query. If you find that a query uses a broadcast join when you know through benchmarking that a
+      partitioned join would be more efficient, or vice versa, add a hint to the query to specify the precise join
+      mechanism to use. See <xref href="impala_hints.xml#hints"/> for details.
+    </p>
+  </conbody>
+
+  <concept rev="1.2.2" id="joins_no_stats">
+
+    <title>How Joins Are Processed when Statistics Are Unavailable</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Concepts"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        If table or column statistics are not available for some tables in a join, Impala still reorders the tables
+        using the information that is available. Tables with statistics are placed on the left side of the join
+        order, in descending order of cost based on overall size and cardinality. Tables without statistics are
+        treated as zero-size, that is, they are always placed on the right side of the join order.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept rev="1.2.2" id="straight_join">
+
+    <title>Overriding Join Reordering with STRAIGHT_JOIN</title>
+
+    <conbody>
+
+      <p>
+        If an Impala join query is inefficient because of outdated statistics or unexpected data distribution, you
+        can keep Impala from reordering the joined tables by using the <codeph>STRAIGHT_JOIN</codeph> keyword
+        immediately after the <codeph>SELECT</codeph> keyword. The <codeph>STRAIGHT_JOIN</codeph> keyword turns off
+        the reordering of join clauses that Impala does internally, and produces a plan that relies on the join
+        clauses being ordered optimally in the query text. In this case, rewrite the query so that the largest
+        table is on the left, followed by the next largest, and so on until the smallest table is on the right.
+      </p>
+
+      <p>
+        In this example, the subselect from the <codeph>BIG</codeph> table produces a very small result set, but
+        the table might still be treated as if it were the biggest and placed first in the join order. Using
+        <codeph>STRAIGHT_JOIN</codeph> for the last join clause prevents the final table from being reordered,
+        keeping it as the rightmost table in the join order.
+      </p>
+
+<codeblock>select straight_join x from medium join small join (select * from big where c1 &lt; 10) as big
+  where medium.id = small.id and small.id = big.id;</codeblock>
+    </conbody>
+  </concept>
+
+  <concept id="perf_joins_examples">
+
+    <title>Examples of Join Order Optimization</title>
+
+    <conbody>
+
+      <p>
+        Here are examples showing joins between tables with 1 billion, 200 million, and 1 million rows. (In this
+        case, the tables are unpartitioned and using Parquet format.) The smaller tables contain subsets of data
+        from the largest one, for convenience of joining on the unique <codeph>ID</codeph> column. The smallest
+        table only contains a subset of columns from the others.
+      </p>
+
+      <p></p>
+
+<codeblock>[localhost:21000] &gt; create table big stored as parquet as select * from raw_data;
++----------------------------+
+| summary                    |
++----------------------------+
+| Inserted 1000000000 row(s) |
++----------------------------+
+Returned 1 row(s) in 671.56s
+[localhost:21000] &gt; desc big;
++-----------+---------+---------+
+| name      | type    | comment |
++-----------+---------+---------+
+| id        | int     |         |
+| val       | int     |         |
+| zfill     | string  |         |
+| name      | string  |         |
+| assertion | boolean |         |
++-----------+---------+---------+
+Returned 5 row(s) in 0.01s
+[localhost:21000] &gt; create table medium stored as parquet as select * from big limit 200 * floor(1e6);
++---------------------------+
+| summary                   |
++---------------------------+
+| Inserted 200000000 row(s) |
++---------------------------+
+Returned 1 row(s) in 138.31s
+[localhost:21000] &gt; create table small stored as parquet as select id,val,name from big where assertion = true limit 1 * floor(1e6);
++-------------------------+
+| summary                 |
++-------------------------+
+| Inserted 1000000 row(s) |
++-------------------------+
+Returned 1 row(s) in 6.32s</codeblock>
+
+      <p>
+        For any kind of performance experimentation, use the <codeph>EXPLAIN</codeph> statement to see how any
+        expensive query will be performed without actually running it, and enable verbose <codeph>EXPLAIN</codeph>
+        plans containing more performance-oriented detail: The most interesting plan lines are highlighted in bold,
+        showing that without statistics for the joined tables, Impala cannot make a good estimate of the number of
+        rows involved at each stage of processing, and is likely to stick with the <codeph>BROADCAST</codeph> join
+        mechanism that sends a complete copy of one of the tables to each node.
+      </p>
+
+<codeblock>[localhost:21000] &gt; set explain_level=verbose;
+EXPLAIN_LEVEL set to verbose
+[localhost:21000] &gt; explain select count(*) from big join medium where big.id = medium.id;
++----------------------------------------------------------+
+| Explain String                                           |
++----------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=2.10GB VCores=2  |
+|                                                          |
+| PLAN FRAGMENT 0                                          |
+|   PARTITION: UNPARTITIONED                               |
+|                                                          |
+|   6:AGGREGATE (merge finalize)                           |
+|   |  output: SUM(COUNT(*))                               |
+|   |  cardinality: 1                                      |
+|   |  per-host memory: unavailable                        |
+|   |  tuple ids: 2                                        |
+|   |                                                      |
+|   5:EXCHANGE                                             |
+|      cardinality: 1                                      |
+|      per-host memory: unavailable                        |
+|      tuple ids: 2                                        |
+|                                                          |
+| PLAN FRAGMENT 1                                          |
+|   PARTITION: RANDOM                                      |
+|                                                          |
+|   STREAM DATA SINK                                       |
+|     EXCHANGE ID: 5                                       |
+|     UNPARTITIONED                                        |
+|                                                          |
+|   3:AGGREGATE                                            |
+|   |  output: COUNT(*)                                    |
+|   |  cardinality: 1                                      |
+|   |  per-host memory: 10.00MB                            |
+|   |  tuple ids: 2                                        |
+|   |                                                      |
+|   2:HASH JOIN                                            |
+<b>|   |  join op: INNER JOIN (BROADCAST)                     |</b>
+|   |  hash predicates:                                    |
+|   |    big.id = medium.id                                |
+<b>|   |  cardinality: unavailable                            |</b>
+|   |  per-host memory: 2.00GB                             |
+|   |  tuple ids: 0 1                                      |
+|   |                                                      |
+|   |----4:EXCHANGE                                        |
+|   |       cardinality: unavailable                       |
+|   |       per-host memory: 0B                            |
+|   |       tuple ids: 1                                   |
+|   |                                                      |
+|   0:SCAN HDFS                                            |
+<b>|      table=join_order.big #partitions=1/1 size=23.12GB   |
+|      table stats: unavailable                            |
+|      column stats: unavailable                           |
+|      cardinality: unavailable                            |</b>
+|      per-host memory: 88.00MB                            |
+|      tuple ids: 0                                        |
+|                                                          |
+| PLAN FRAGMENT 2                                          |
+|   PARTITION: RANDOM                                      |
+|                                                          |
+|   STREAM DATA SINK                                       |
+|     EXCHANGE ID: 4                                       |
+|     UNPARTITIONED                                        |
+|                                                          |
+|   1:SCAN HDFS                                            |
+<b>|      table=join_order.medium #partitions=1/1 size=4.62GB |
+|      table stats: unavailable                            |
+|      column stats: unavailable                           |
+|      cardinality: unavailable                            |</b>
+|      per-host memory: 88.00MB                            |
+|      tuple ids: 1                                        |
++----------------------------------------------------------+
+Returned 64 row(s) in 0.04s</codeblock>
+
+      <p>
+        Gathering statistics for all the tables is straightforward, one <codeph>COMPUTE STATS</codeph> statement
+        per table:
+      </p>
+
+<codeblock>[localhost:21000] &gt; compute stats small;
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 1 partition(s) and 3 column(s). |
++-----------------------------------------+
+Returned 1 row(s) in 4.26s
+[localhost:21000] &gt; compute stats medium;
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 1 partition(s) and 5 column(s). |
++-----------------------------------------+
+Returned 1 row(s) in 42.11s
+[localhost:21000] &gt; compute stats big;
++-----------------------------------------+
+| summary                                 |
++-----------------------------------------+
+| Updated 1 partition(s) and 5 column(s). |
++-----------------------------------------+
+Returned 1 row(s) in 165.44s</codeblock>
+
+      <p>
+        With statistics in place, Impala can choose a more effective join order rather than following the
+        left-to-right sequence of tables in the query, and can choose <codeph>BROADCAST</codeph> or
+        <codeph>PARTITIONED</codeph> join strategies based on the overall sizes and number of rows in the table:
+      </p>
+
+<codeblock>[localhost:21000] &gt; explain select count(*) from medium join big where big.id = medium.id;
+Query: explain select count(*) from medium join big where big.id = medium.id
++-----------------------------------------------------------+
+| Explain String                                            |
++-----------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=937.23MB VCores=2 |
+|                                                           |
+| PLAN FRAGMENT 0                                           |
+|   PARTITION: UNPARTITIONED                                |
+|                                                           |
+|   6:AGGREGATE (merge finalize)                            |
+|   |  output: SUM(COUNT(*))                                |
+|   |  cardinality: 1                                       |
+|   |  per-host memory: unavailable                         |
+|   |  tuple ids: 2                                         |
+|   |                                                       |
+|   5:EXCHANGE                                              |
+|      cardinality: 1                                       |
+|      per-host memory: unavailable                         |
+|      tuple ids: 2                                         |
+|                                                           |
+| PLAN FRAGMENT 1                                           |
+|   PARTITION: RANDOM                                       |
+|                                                           |
+|   STREAM DATA SINK                                        |
+|     EXCHANGE ID: 5                                        |
+|     UNPARTITIONED                                         |
+|                                                           |
+|   3:AGGREGATE                                             |
+|   |  output: COUNT(*)                                     |
+|   |  cardinality: 1                                       |
+|   |  per-host memory: 10.00MB                             |
+|   |  tuple ids: 2                                         |
+|   |                                                       |
+|   2:HASH JOIN                                             |
+|   |  join op: INNER JOIN (BROADCAST)                      |
+|   |  hash predicates:                                     |
+|   |    big.id = medium.id                                 |
+|   |  cardinality: 1443004441                              |
+|   |  per-host memory: 839.23MB                            |
+|   |  tuple ids: 1 0                                       |
+|   |                                                       |
+|   |----4:EXCHANGE                                         |
+|   |       cardinality: 200000000                          |
+|   |       per-host memory: 0B                             |
+|   |       tuple ids: 0                                    |
+|   |                                                       |
+|   1:SCAN HDFS                                             |
+|      table=join_order.big #partitions=1/1 size=23.12GB    |
+|      table stats: 1000000000 rows total                   |
+|      column stats: all                                    |
+|      cardinality: 1000000000                              |
+|      per-host memory: 88.00MB                             |
+|      tuple ids: 1                                         |
+|                                                           |
+| PLAN FRAGMENT 2                                           |
+|   PARTITION: RANDOM                                       |
+|                                                           |
+|   STREAM DATA SINK                                        |
+|     EXCHANGE ID: 4                                        |
+|     UNPARTITIONED                                         |
+|                                                           |
+|   0:SCAN HDFS                                             |
+|      table=join_order.medium #partitions=1/1 size=4.62GB  |
+|      table stats: 200000000 rows total                    |
+|      column stats: all                                    |
+|      cardinality: 200000000                               |
+|      per-host memory: 88.00MB                             |
+|      tuple ids: 0                                         |
++-----------------------------------------------------------+
+Returned 64 row(s) in 0.04s
+
+[localhost:21000] &gt; explain select count(*) from small join big where big.id = small.id;
+Query: explain select count(*) from small join big where big.id = small.id
++-----------------------------------------------------------+
+| Explain String                                            |
++-----------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=101.15MB VCores=2 |
+|                                                           |
+| PLAN FRAGMENT 0                                           |
+|   PARTITION: UNPARTITIONED                                |
+|                                                           |
+|   6:AGGREGATE (merge finalize)                            |
+|   |  output: SUM(COUNT(*))                                |
+|   |  cardinality: 1                                       |
+|   |  per-host memory: unavailable                         |
+|   |  tuple ids: 2                                         |
+|   |                                                       |
+|   5:EXCHANGE                                              |
+|      cardinality: 1                                       |
+|      per-host memory: unavailable                         |
+|      tuple ids: 2                                         |
+|                                                           |
+| PLAN FRAGMENT 1                                           |
+|   PARTITION: RANDOM                                       |
+|                                                           |
+|   STREAM DATA SINK                                        |
+|     EXCHANGE ID: 5                                        |
+|     UNPARTITIONED                                         |
+|                                                           |
+|   3:AGGREGATE                                             |
+|   |  output: COUNT(*)                                     |
+|   |  cardinality: 1                                       |
+|   |  per-host memory: 10.00MB                             |
+|   |  tuple ids: 2                                         |
+|   |                                                       |
+|   2:HASH JOIN                                             |
+|   |  join op: INNER JOIN (BROADCAST)                      |
+|   |  hash predicates:                                     |
+|   |    big.id = small.id                                  |
+|   |  cardinality: 1000000000                              |
+|   |  per-host memory: 3.15MB                              |
+|   |  tuple ids: 1 0                                       |
+|   |                                                       |
+|   |----4:EXCHANGE                                         |
+|   |       cardinality: 1000000                            |
+|   |       per-host memory: 0B                             |
+|   |       tuple ids: 0                                    |
+|   |                                                       |
+|   1:SCAN HDFS                                             |
+|      table=join_order.big #partitions=1/1 size=23.12GB    |
+|      table stats: 1000000000 rows total                   |
+|      column stats: all                                    |
+|      cardinality: 1000000000                              |
+|      per-host memory: 88.00MB                             |
+|      tuple ids: 1                                         |
+|                                                           |
+| PLAN FRAGMENT 2                                           |
+|   PARTITION: RANDOM                                       |
+|                                                           |
+|   STREAM DATA SINK                                        |
+|     EXCHANGE ID: 4                                        |
+|     UNPARTITIONED                                         |
+|                                                           |
+|   0:SCAN HDFS                                             |
+|      table=join_order.small #partitions=1/1 size=17.93MB  |
+|      table stats: 1000000 rows total                      |
+|      column stats: all                                    |
+|      cardinality: 1000000                                 |
+|      per-host memory: 32.00MB                             |
+|      tuple ids: 0                                         |
++-----------------------------------------------------------+
+Returned 64 row(s) in 0.03s</codeblock>
+
+      <p>
+        When queries like these are actually run, the execution times are relatively consistent regardless of the
+        table order in the query text. Here are examples using both the unique <codeph>ID</codeph> column and the
+        <codeph>VAL</codeph> column containing duplicate values:
+      </p>
+
+<codeblock>[localhost:21000] &gt; select count(*) from big join small on (big.id = small.id);
+Query: select count(*) from big join small on (big.id = small.id)
++----------+
+| count(*) |
++----------+
+| 1000000  |
++----------+
+Returned 1 row(s) in 21.68s
+[localhost:21000] &gt; select count(*) from small join big on (big.id = small.id);
+Query: select count(*) from small join big on (big.id = small.id)
++----------+
+| count(*) |
++----------+
+| 1000000  |
++----------+
+Returned 1 row(s) in 20.45s
+
+[localhost:21000] &gt; select count(*) from big join small on (big.val = small.val);
++------------+
+| count(*)   |
++------------+
+| 2000948962 |
++------------+
+Returned 1 row(s) in 108.85s
+[localhost:21000] &gt; select count(*) from small join big on (big.val = small.val);
++------------+
+| count(*)   |
++------------+
+| 2000948962 |
++------------+
+Returned 1 row(s) in 100.76s</codeblock>
+
+      <note>
+        When examining the performance of join queries and the effectiveness of the join order optimization, make
+        sure the query involves enough data and cluster resources to see a difference depending on the query plan.
+        For example, a single data file of just a few megabytes will reside in a single HDFS block and be processed
+        on a single node. Likewise, if you use a single-node or two-node cluster, there might not be much
+        difference in efficiency for the broadcast or partitioned join strategies.
+      </note>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_perf_resources.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_perf_resources.xml b/docs/topics/impala_perf_resources.xml
new file mode 100644
index 0000000..c538893
--- /dev/null
+++ b/docs/topics/impala_perf_resources.xml
@@ -0,0 +1,60 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="mem_limits">
+
+  <title>Controlling Impala Resource Usage</title>
+  <titlealts audience="PDF"><navtitle>Controlling Resource Usage</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Memory"/>
+      <data name="Category" value="Scalability"/>
+      <data name="Category" value="Resource Management"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Sometimes, balancing raw query performance against scalability requires limiting the amount of resources,
+      such as memory or CPU, used by a single query or group of queries. Impala can use several mechanisms that
+      help to smooth out the load during heavy concurrent usage, resulting in faster overall query times and
+      sharing of resources across Impala queries, MapReduce jobs, and other kinds of workloads across a CDH
+      cluster:
+    </p>
+
+    <ul>
+      <li>
+        The Impala admission control feature uses a fast, distributed mechanism to hold back queries that exceed
+        limits on the number of concurrent queries or the amount of memory used. The queries are queued, and
+        executed as other queries finish and resources become available. You can control the concurrency limits,
+        and specify different limits for different groups of users to divide cluster resources according to the
+        priorities of different classes of users. This feature is new in Impala 1.3, and works with both CDH 4 and
+        CDH 5. See <xref href="impala_admission.xml#admission_control"/> for details.
+      </li>
+
+      <li>
+        <p>
+          You can restrict the amount of memory Impala reserves during query execution by specifying the
+          <codeph>-mem_limit</codeph> option for the <codeph>impalad</codeph> daemon. See
+          <xref href="impala_config_options.xml#config_options"/> for details. This limit applies only to the
+          memory that is directly consumed by queries; Impala reserves additional memory at startup, for example to
+          hold cached metadata.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          For production deployment, <ph rev="upstream">Cloudera</ph> recommends that you implement resource isolation using mechanisms
+          such as cgroups, which you can configure using Cloudera Manager. For details, see the
+          <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_service_pools.html" scope="external" format="html">Static
+          Resource Pools</xref> in the Cloudera Manager documentation.
+        </p>
+      </li>
+    </ul>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_perf_skew.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_perf_skew.xml b/docs/topics/impala_perf_skew.xml
new file mode 100644
index 0000000..b3a7cec
--- /dev/null
+++ b/docs/topics/impala_perf_skew.xml
@@ -0,0 +1,150 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="perf_skew">
+
+  <title>Detecting and Correcting HDFS Block Skew Conditions</title>
+  <titlealts audience="PDF"><navtitle>HDFS Block Skew</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="HDFS"/>
+      <data name="Category" value="Proof of Concept"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      For best performance of Impala parallel queries, the work is divided equally across hosts in the cluster, and
+      all hosts take approximately equal time to finish their work. If one host takes substantially longer than
+      others, the extra time needed for the slow host can become the dominant factor in query performance.
+      Therefore, one of the first steps in performance tuning for Impala is to detect and correct such conditions.
+    </p>
+
+    <p>
+      The main cause of uneven performance that you can correct within Impala is <term>skew</term> in the number of
+      HDFS data blocks processed by each host, where some hosts process substantially more data blocks than others.
+      This condition can occur because of uneven distribution of the data values themselves, for example causing
+      certain data files or partitions to be large while others are very small. (Although it is possible to have
+      unevenly distributed data without any problems with the distribution of HDFS blocks.) Block skew could also
+      be due to the underlying block allocation policies within HDFS, the replication factor of the data files, and
+      the way that Impala chooses the host to process each data block.
+    </p>
+
+    <p>
+      The most convenient way to detect block skew, or slow-host issues in general, is to examine the <q>executive
+      summary</q> information from the query profile after running a query:
+    </p>
+
+    <ul>
+      <li>
+        <p>
+          In <cmdname>impala-shell</cmdname>, issue the <codeph>SUMMARY</codeph> command immediately after the
+          query is complete, to see just the summary information. If you detect issues involving skew, you might
+          switch to issuing the <codeph>PROFILE</codeph> command, which displays the summary information followed
+          by a detailed performance analysis.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          In the Cloudera Manager interface or the Impala debug web UI, click on the <uicontrol>Profile</uicontrol>
+          link associated with the query after it is complete. The executive summary information is displayed early
+          in the profile output.
+        </p>
+      </li>
+    </ul>
+
+    <p>
+      For each phase of the query, you see an <uicontrol>Avg Time</uicontrol> and a <uicontrol>Max Time</uicontrol>
+      value, along with <uicontrol>#Hosts</uicontrol> indicating how many hosts are involved in that query phase.
+      For all the phases with <uicontrol>#Hosts</uicontrol> greater than one, look for cases where the maximum time
+      is substantially greater than the average time. Focus on the phases that took the longest, for example, those
+      taking multiple seconds rather than milliseconds or microseconds.
+    </p>
+
+    <p>
+      If you detect that some hosts take longer than others, first rule out non-Impala causes. One reason that some
+      hosts could be slower than others is if those hosts have less capacity than the others, or if they are
+      substantially busier due to unevenly distributed non-Impala workloads:
+    </p>
+
+    <ul>
+      <li>
+        <p>
+          For clusters running Impala, keep the relative capacities of all hosts roughly equal. Any cost savings
+          from including some underpowered hosts in the cluster will likely be outweighed by poor or uneven
+          performance, and the time spent diagnosing performance issues.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          If non-Impala workloads cause slowdowns on some hosts but not others, use the appropriate load-balancing
+          techniques for the non-Impala components to smooth out the load across the cluster.
+        </p>
+      </li>
+    </ul>
+
+    <p>
+      If the hosts on your cluster are evenly powered and evenly loaded, examine the detailed profile output to
+      determine which host is taking longer than others for the query phase in question. Examine how many bytes are
+      processed during that phase on that host, how much memory is used, and how many bytes are transmitted across
+      the network.
+    </p>
+
+    <p>
+      The most common symptom is a higher number of bytes read on one host than others, due to one host being
+      requested to process a higher number of HDFS data blocks. This condition is more likely to occur when the
+      number of blocks accessed by the query is relatively small. For example, if you have a 10-node cluster and
+      the query processes 10 HDFS blocks, each node might not process exactly one block. If one node sits idle
+      while another node processes two blocks, the query could take twice as long as if the data was perfectly
+      distributed.
+    </p>
+
+    <p>
+      Possible solutions in this case include:
+    </p>
+
+    <ul>
+      <li>
+        <p>
+          If the query is artificially small, perhaps for benchmarking purposes, scale it up to process a larger
+          data set. For example, if some nodes read 10 HDFS data blocks while others read 11, the overall effect of
+          the uneven distribution is much lower than when some nodes did twice as much work as others. As a
+          guideline, aim for a <q>sweet spot</q> where each node reads 2 GB or more from HDFS per query. Queries
+          that process lower volumes than that could experience inconsistent performance that smooths out as
+          queries become more data-intensive.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          If the query processes only a few large blocks, so that many nodes sit idle and cannot help to
+          parallelize the query, consider reducing the overall block size. For example, you might adjust the
+          <codeph>PARQUET_FILE_SIZE</codeph> query option before copying or converting data into a Parquet table.
+          Or you might adjust the granularity of data files produced earlier in the ETL pipeline by non-Impala
+          components. In Impala 2.0 and later, the default Parquet block size is 256 MB, reduced from 1 GB, to
+          improve parallelism for common cluster sizes and data volumes.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          Reduce the amount of compression applied to the data. For text data files, the highest degree of
+          compression (gzip) produces unsplittable files that are more difficult for Impala to process in parallel,
+          and require extra memory during processing to hold the compressed and uncompressed data simultaneously.
+          For binary formats such as Parquet and Avro, compression can result in fewer data blocks overall, but
+          remember that when queries process relatively few blocks, there is less opportunity for parallel
+          execution and many nodes in the cluster might sit idle. Note that when Impala writes Parquet data with
+          the query option <codeph>COMPRESSION_CODEC=NONE</codeph> enabled, the data is still typically compact due
+          to the encoding schemes used by Parquet, independent of the final compression step.
+        </p>
+      </li>
+    </ul>
+  </conbody>
+</concept>



[16/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_optimize_partition_key_scans.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_optimize_partition_key_scans.xml b/docs/topics/impala_optimize_partition_key_scans.xml
new file mode 100644
index 0000000..2cae1cf
--- /dev/null
+++ b/docs/topics/impala_optimize_partition_key_scans.xml
@@ -0,0 +1,180 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="2.5.0 IMPALA-2499" id="optimize_partition_key_scans">
+
+  <title>OPTIMIZE_PARTITION_KEY_SCANS Query Option (<keyword keyref="impala25"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>OPTIMIZE_PARTITION_KEY_SCANS</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.5.0 IMPALA-2499">
+      <indexterm audience="Cloudera">OPTIMIZE_PARTITION_KEY_SCANS query option</indexterm>
+      Enables a fast code path for queries that apply simple aggregate functions to partition key
+      columns: <codeph>MIN(<varname>key_column</varname>)</codeph>, <codeph>MAX(<varname>key_column</varname>)</codeph>,
+      or <codeph>COUNT(DISTINCT <varname>key_column</varname>)</codeph>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_boolean"/>
+    <p conref="../shared/impala_common.xml#common/default_false_0"/>
+
+    <note conref="../shared/impala_common.xml#common/one_but_not_true"/>
+
+    <p conref="../shared/impala_common.xml#common/added_in_250"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      This optimization speeds up common <q>introspection</q> operations when using queries
+      to calculate the cardinality and range for partition key columns.
+    </p>
+
+    <p>
+      This optimization does not apply if the queries contain any <codeph>WHERE</codeph>,
+      <codeph>GROUP BY</codeph>, or <codeph>HAVING</codeph> clause. The relevant queries
+      should only compute the minimum, maximum, or number of distinct values for the
+      partition key columns across the whole table.
+    </p>
+
+    <p>
+      This optimization is enabled by a query option because it skips some consistency checks
+      and therefore can return slightly different partition values if partitions are in the
+      process of being added, dropped, or loaded outside of Impala. Queries might exhibit different
+      behavior depending on the setting of this option in the following cases:
+    </p>
+
+    <ul>
+      <li>
+        <p>
+          If files are removed from a partition using HDFS or other non-Impala operations,
+          there is a period until the next <codeph>REFRESH</codeph> of the table where regular
+          queries fail at run time because they detect the missing files. With this optimization
+          enabled, queries that evaluate only the partition key column values (not the contents of
+          the partition itself) succeed, and treat the partition as if it still exists.
+        </p>
+      </li>
+      <li>
+        <p>
+          If a partition contains any data files, but the data files do not contain any rows,
+          a regular query considers that the partition does not exist. With this optimization
+          enabled, the partition is treated as if it exists.
+        </p>
+        <p>
+          If the partition includes no files at all, this optimization does not change the query
+          behavior: the partition is considered to not exist whether or not this optimization is enabled.
+        </p>
+      </li>
+    </ul>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      The following example shows initial schema setup and the default behavior of queries that
+      return just the partition key column for a table:
+    </p>
+
+<codeblock>
+-- Make a partitioned table with 3 partitions.
+create table t1 (s string) partitioned by (year int);
+insert into t1 partition (year=2015) values ('last year');
+insert into t1 partition (year=2016) values ('this year');
+insert into t1 partition (year=2017) values ('next year');
+
+-- Regardless of the option setting, this query must read the
+-- data files to know how many rows to return for each year value.
+explain select year from t1;
++-----------------------------------------------------+
+| Explain String                                      |
++-----------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=0B VCores=0 |
+|                                                     |
+| F00:PLAN FRAGMENT [UNPARTITIONED]                   |
+|   00:SCAN HDFS [key_cols.t1]                        |
+|      partitions=3/3 files=4 size=40B                |
+|      table stats: 3 rows total                      |
+|      column stats: all                              |
+|      hosts=3 per-host-mem=unavailable               |
+|      tuple-ids=0 row-size=4B cardinality=3          |
++-----------------------------------------------------+
+
+-- The aggregation operation means the query does not need to read
+-- the data within each partition: the result set contains exactly 1 row
+-- per partition, derived from the partition key column value.
+-- By default, Impala still includes a 'scan' operation in the query.
+explain select distinct year from t1;
++------------------------------------------------------------------------------------+
+| Explain String                                                                     |
++------------------------------------------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=0B VCores=0                                |
+|                                                                                    |
+| 01:AGGREGATE [FINALIZE]                                                            |
+| |  group by: year                                                                  |
+| |                                                                                  |
+| 00:SCAN HDFS [key_cols.t1]                                                         |
+|    partitions=0/0 files=0 size=0B                                                  |
++------------------------------------------------------------------------------------+
+</codeblock>
+
+    <p>
+      The following examples show how the plan is made more efficient when the
+      <codeph>OPTIMIZE_PARTITION_KEY_SCANS</codeph> option is enabled:
+    </p>
+
+<codeblock>
+set optimize_partition_key_scans=1;
+OPTIMIZE_PARTITION_KEY_SCANS set to 1
+
+-- The aggregation operation is turned into a UNION internally,
+-- with constant values known in advance based on the metadata
+-- for the partitioned table.
+explain select distinct year from t1;
++-----------------------------------------------------+
+| Explain String                                      |
++-----------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=0B VCores=0 |
+|                                                     |
+| F00:PLAN FRAGMENT [UNPARTITIONED]                   |
+|   01:AGGREGATE [FINALIZE]                           |
+|   |  group by: year                                 |
+|   |  hosts=1 per-host-mem=unavailable               |
+|   |  tuple-ids=1 row-size=4B cardinality=3          |
+|   |                                                 |
+|   00:UNION                                          |
+|      constant-operands=3                            |
+|      hosts=1 per-host-mem=unavailable               |
+|      tuple-ids=0 row-size=4B cardinality=3          |
++-----------------------------------------------------+
+
+-- The same optimization applies to other aggregation queries
+-- that only return values based on partition key columns:
+-- MIN, MAX, COUNT(DISTINCT), and so on.
+explain select min(year) from t1;
++-----------------------------------------------------+
+| Explain String                                      |
++-----------------------------------------------------+
+| Estimated Per-Host Requirements: Memory=0B VCores=0 |
+|                                                     |
+| F00:PLAN FRAGMENT [UNPARTITIONED]                   |
+|   01:AGGREGATE [FINALIZE]                           |
+|   |  output: min(year)                              |
+|   |  hosts=1 per-host-mem=unavailable               |
+|   |  tuple-ids=1 row-size=4B cardinality=1          |
+|   |                                                 |
+|   00:UNION                                          |
+|      constant-operands=3                            |
+|      hosts=1 per-host-mem=unavailable               |
+|      tuple-ids=0 row-size=4B cardinality=3          |
++-----------------------------------------------------+
+</codeblock>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_order_by.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_order_by.xml b/docs/topics/impala_order_by.xml
new file mode 100644
index 0000000..2db141d
--- /dev/null
+++ b/docs/topics/impala_order_by.xml
@@ -0,0 +1,318 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="order_by">
+
+  <title>ORDER BY Clause</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      The familiar <codeph>ORDER BY</codeph> clause of a <codeph>SELECT</codeph> statement sorts the result set
+      based on the values from one or more columns.
+    </p>
+
+    <p>
+      For distributed queries, this is a relatively expensive operation, because the entire result set must be
+      produced and transferred to one node before the sorting can happen. This can require more memory capacity
+      than a query without <codeph>ORDER BY</codeph>. Even if the query takes approximately the same time to finish
+      with or without the <codeph>ORDER BY</codeph> clause, subjectively it can appear slower because no results
+      are available until all processing is finished, rather than results coming back gradually as rows matching
+      the <codeph>WHERE</codeph> clause are found. Therefore, if you only need the first N results from the sorted
+      result set, also include the <codeph>LIMIT</codeph> clause, which reduces network overhead and the memory
+      requirement on the coordinator node.
+    </p>
+
+    <note>
+      <p rev="1.4.0 obwl">
+        In Impala 1.4.0 and higher, the <codeph>LIMIT</codeph> clause is now optional (rather than required) for
+        queries that use the <codeph>ORDER BY</codeph> clause. Impala automatically uses a temporary disk work area
+        to perform the sort if the sort operation would otherwise exceed the Impala memory limit for a particular
+        DataNode.
+      </p>
+    </note>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p>
+      The full syntax for the <codeph>ORDER BY</codeph> clause is:
+    </p>
+
+<codeblock rev="1.2.1">ORDER BY <varname>col_ref</varname> [, <varname>col_ref</varname> ...] [ASC | DESC] [NULLS FIRST | NULLS LAST]
+
+col_ref ::= <varname>column_name</varname> | <varname>integer_literal</varname>
+</codeblock>
+
+    <p>
+      Although the most common usage is <codeph>ORDER BY <varname>column_name</varname></codeph>, you can also
+      specify <codeph>ORDER BY 1</codeph> to sort by the first column of the result set, <codeph>ORDER BY
+      2</codeph> to sort by the second column, and so on. The number must be a numeric literal, not some other kind
+      of constant expression. (If the argument is some other expression, even a <codeph>STRING</codeph> value, the
+      query succeeds but the order of results is undefined.)
+    </p>
+
+    <p>
+      <codeph>ORDER BY <varname>column_number</varname></codeph> can only be used when the query explicitly lists
+      the columns in the <codeph>SELECT</codeph> list, not with <codeph>SELECT *</codeph> queries.
+    </p>
+
+    <p>
+      <b>Ascending and descending sorts:</b>
+    </p>
+
+    <p>
+      The default sort order (the same as using the <codeph>ASC</codeph> keyword) puts the smallest values at the
+      start of the result set, and the largest values at the end. Specifying the <codeph>DESC</codeph> keyword
+      reverses that order.
+    </p>
+
+    <p>
+      <b>Sort order for NULL values:</b>
+    </p>
+
+    <p rev="1.2.1">
+      See <xref href="impala_literals.xml#null"/> for details about how <codeph>NULL</codeph> values are positioned
+      in the sorted result set, and how to use the <codeph>NULLS FIRST</codeph> and <codeph>NULLS LAST</codeph>
+      clauses. (The sort position for <codeph>NULL</codeph> values in <codeph>ORDER BY ... DESC</codeph> queries is
+      changed in Impala 1.2.1 and higher to be more standards-compliant, and the <codeph>NULLS FIRST</codeph> and
+      <codeph>NULLS LAST</codeph> keywords are new in Impala 1.2.1.)
+    </p>
+
+    <p rev="obwl" conref="../shared/impala_common.xml#common/order_by_limit"/>
+
+    <!-- Good to show an example of cases where ORDER BY does and doesn't work with complex types. -->
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+    <p rev="2.3.0">
+      In <keyword keyref="impala23_full"/> and higher, the complex data types <codeph>STRUCT</codeph>,
+      <codeph>ARRAY</codeph>, and <codeph>MAP</codeph> are available. These columns cannot
+      be referenced directly in the <codeph>ORDER BY</codeph> clause.
+      When you query a complex type column, you use join notation to <q>unpack</q> the elements
+      of the complex type, and within the join query you can include an <codeph>ORDER BY</codeph>
+      clause to control the order in the result set of the scalar elements from the complex type.
+      See <xref href="impala_complex_types.xml#complex_types"/> for details about Impala support for complex types.
+    </p>
+
+    <p>
+      The following query shows how a complex type column cannot be directly used in an <codeph>ORDER BY</codeph> clause:
+    </p>
+
+<codeblock>CREATE TABLE games (id BIGINT, score ARRAY &lt;BIGINT&gt;) STORED AS PARQUET;
+...use LOAD DATA to load externally created Parquet files into the table...
+SELECT id FROM games ORDER BY score DESC;
+ERROR: AnalysisException: ORDER BY expression 'score' with complex type 'ARRAY&lt;BIGINT&gt;' is not supported.
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      The following query retrieves the user ID and score, only for scores greater than one million,
+      with the highest scores for each user listed first.
+      Because the individual array elements are now represented as separate rows in the result set,
+      they can be used in the <codeph>ORDER BY</codeph> clause, referenced using the <codeph>ITEM</codeph>
+      pseudocolumn that represents each array element.
+    </p>
+
+<codeblock>SELECT id, item FROM games, games.score
+  WHERE item &gt; 1000000
+ORDER BY id, item desc;
+</codeblock>
+
+    <p>
+      The following queries use similar <codeph>ORDER BY</codeph> techniques with variations of the <codeph>GAMES</codeph>
+      table, where the complex type is an <codeph>ARRAY</codeph> containing <codeph>STRUCT</codeph> or <codeph>MAP</codeph>
+      elements to represent additional details about each game that was played.
+      For an array of structures, the fields of the structure are referenced as <codeph>ITEM.<varname>field_name</varname></codeph>.
+      For an array of maps, the keys and values within each array element are referenced as <codeph>ITEM.KEY</codeph>
+      and <codeph>ITEM.VALUE</codeph>.
+    </p>
+
+<codeblock>CREATE TABLE games2 (id BIGINT, play array &lt; struct &lt;game_name: string, score: BIGINT, high_score: boolean&gt; &gt;) STORED AS PARQUET
+...use LOAD DATA to load externally created Parquet files into the table...
+SELECT id, item.game_name, item.score FROM games2, games2.play
+  WHERE item.score &gt; 1000000
+ORDER BY id, item.score DESC;
+
+CREATE TABLE games3 (id BIGINT, play ARRAY &lt; MAP &lt;STRING, BIGINT&gt; &gt;) STORED AS PARQUET;
+...use LOAD DATA to load externally created Parquet files into the table...
+SELECT id, info.key AS k, info.value AS v from games3, games3.play AS plays, games3.play.item AS info
+  WHERE info.KEY = 'score' AND info.VALUE &gt; 1000000
+ORDER BY id, info.value desc;
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      Although the <codeph>LIMIT</codeph> clause is now optional on <codeph>ORDER BY</codeph> queries, if your
+      query only needs some number of rows that you can predict in advance, use the <codeph>LIMIT</codeph> clause
+      to reduce unnecessary processing. For example, if the query has a clause <codeph>LIMIT 10</codeph>, each data
+      node sorts its portion of the relevant result set and only returns 10 rows to the coordinator node. The
+      coordinator node picks the 10 highest or lowest row values out of this small intermediate result set.
+    </p>
+
+    <p>
+      If an <codeph>ORDER BY</codeph> clause is applied to an early phase of query processing, such as a subquery
+      or a view definition, Impala ignores the <codeph>ORDER BY</codeph> clause. To get ordered results from a
+      subquery or view, apply an <codeph>ORDER BY</codeph> clause to the outermost or final <codeph>SELECT</codeph>
+      level.
+    </p>
+
+    <p>
+      <codeph>ORDER BY</codeph> is often used in combination with <codeph>LIMIT</codeph> to perform <q>top-N</q>
+      queries:
+    </p>
+
+<codeblock>SELECT user_id AS "Top 10 Visitors", SUM(page_views) FROM web_stats
+  GROUP BY page_views, user_id
+  ORDER BY SUM(page_views) DESC LIMIT 10;
+</codeblock>
+
+    <p>
+      <codeph>ORDER BY</codeph> is sometimes used in combination with <codeph>OFFSET</codeph> and
+      <codeph>LIMIT</codeph> to paginate query results, although it is relatively inefficient to issue multiple
+      queries like this against the large tables typically used with Impala:
+    </p>
+
+<codeblock>SELECT page_title AS "Page 1 of search results", page_url FROM search_content
+  WHERE LOWER(page_title) LIKE '%game%')
+  ORDER BY page_title LIMIT 10 OFFSET 0;
+SELECT page_title AS "Page 2 of search results", page_url FROM search_content
+  WHERE LOWER(page_title) LIKE '%game%')
+  ORDER BY page_title LIMIT 10 OFFSET 10;
+SELECT page_title AS "Page 3 of search results", page_url FROM search_content
+  WHERE LOWER(page_title) LIKE '%game%')
+  ORDER BY page_title LIMIT 10 OFFSET 20;
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/internals_blurb"/>
+
+    <p>
+      Impala sorts the intermediate results of an <codeph>ORDER BY</codeph> clause in memory whenever practical. In
+      a cluster of N DataNodes, each node sorts roughly 1/Nth of the result set, the exact proportion varying
+      depending on how the data matching the query is distributed in HDFS.
+    </p>
+
+    <p>
+      If the size of the sorted intermediate result set on any DataNode would cause the query to exceed the Impala
+      memory limit, Impala sorts as much as practical in memory, then writes partially sorted data to disk. (This
+      technique is known in industry terminology as <q>external sorting</q> and <q>spilling to disk</q>.) As each
+      8 MB batch of data is written to disk, Impala frees the corresponding memory to sort a new 8 MB batch of
+      data. When all the data has been processed, a final merge sort operation is performed to correctly order the
+      in-memory and on-disk results as the result set is transmitted back to the coordinator node. When external
+      sorting becomes necessary, Impala requires approximately 60 MB of RAM at a minimum for the buffers needed to
+      read, write, and sort the intermediate results. If more RAM is available on the DataNode, Impala will use
+      the additional RAM to minimize the amount of disk I/O for sorting.
+    </p>
+
+    <p>
+      This external sort technique is used as appropriate on each DataNode (possibly including the coordinator
+      node) to sort the portion of the result set that is processed on that node. When the sorted intermediate
+      results are sent back to the coordinator node to produce the final result set, the coordinator node uses a
+      merge sort technique to produce a final sorted result set without using any extra resources on the
+      coordinator node.
+    </p>
+
+    <p rev="obwl">
+      <b>Configuration for disk usage:</b>
+    </p>
+
+    <p rev="obwl" conref="../shared/impala_common.xml#common/order_by_scratch_dir"/>
+
+<!-- Here is actually the more logical place to collect all those examples, move them from SELECT and cross-reference to here. -->
+
+<!--     <p rev="obwl" conref="../shared/impala_common.xml#common/restrictions_blurb"/> -->
+
+    <p rev="obwl" conref="../shared/impala_common.xml#common/insert_sort_blurb"/>
+
+    <p rev="obwl" conref="../shared/impala_common.xml#common/order_by_view_restriction"/>
+
+    <p>
+      With the lifting of the requirement to include a <codeph>LIMIT</codeph> clause in every <codeph>ORDER
+      BY</codeph> query (in Impala 1.4 and higher):
+    </p>
+
+    <ul>
+      <li>
+        <p>
+          Now the use of scratch disk space raises the possibility of an <q>out of disk space</q> error on a
+          particular DataNode, as opposed to the previous possibility of an <q>out of memory</q> error. Make sure
+          to keep at least 1 GB free on the filesystem used for temporary sorting work.
+        </p>
+      </li>
+
+      <li>
+        <p>
+          The query options
+          <xref href="impala_default_order_by_limit.xml#default_order_by_limit">DEFAULT_ORDER_BY_LIMIT</xref> and
+          <xref href="impala_abort_on_default_limit_exceeded.xml#abort_on_default_limit_exceeded">ABORT_ON_DEFAULT_LIMIT_EXCEEDED</xref>,
+          which formerly controlled the behavior of <codeph>ORDER BY</codeph> queries with no limit specified, are
+          now ignored.
+        </p>
+      </li>
+    </ul>
+
+    <p rev="obwl" conref="../shared/impala_common.xml#common/null_sorting_change"/>
+<codeblock>[localhost:21000] > create table numbers (x int);
+[localhost:21000] > insert into numbers values (1), (null), (2), (null), (3);
+[localhost:21000] > select x from numbers order by x nulls first;
++------+
+| x    |
++------+
+| NULL |
+| NULL |
+| 1    |
+| 2    |
+| 3    |
++------+
+[localhost:21000] > select x from numbers order by x desc nulls first;
++------+
+| x    |
++------+
+| NULL |
+| NULL |
+| 3    |
+| 2    |
+| 1    |
++------+
+[localhost:21000] > select x from numbers order by x nulls last;
++------+
+| x    |
++------+
+| 1    |
+| 2    |
+| 3    |
+| NULL |
+| NULL |
++------+
+[localhost:21000] > select x from numbers order by x desc nulls last;
++------+
+| x    |
++------+
+| 3    |
+| 2    |
+| 1    |
+| NULL |
+| NULL |
++------+
+</codeblock>
+
+    <p rev="obwl" conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p rev="obwl">
+      See <xref href="impala_select.xml#select"/> for further examples of queries with the <codeph>ORDER
+      BY</codeph> clause.
+    </p>
+
+    <p>
+      Analytic functions use the <codeph>ORDER BY</codeph> clause in a different context to define the sequence in
+      which rows are analyzed. See <xref href="impala_analytic_functions.xml#analytic_functions"/> for details.
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_parquet.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_parquet.xml b/docs/topics/impala_parquet.xml
new file mode 100644
index 0000000..b22fa84
--- /dev/null
+++ b/docs/topics/impala_parquet.xml
@@ -0,0 +1,1160 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="parquet">
+
+  <title>Using the Parquet File Format with Impala Tables</title>
+  <titlealts audience="PDF"><navtitle>Parquet Data Files</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="File Formats"/>
+      <data name="Category" value="Parquet"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">Parquet support in Impala</indexterm>
+      Impala helps you to create, manage, and query Parquet tables. Parquet is a column-oriented binary file format
+      intended to be highly efficient for the types of large-scale queries that Impala is best at. Parquet is
+      especially good for queries scanning particular columns within a table, for example to query <q>wide</q>
+      tables with many columns, or to perform aggregation operations such as <codeph>SUM()</codeph> and
+      <codeph>AVG()</codeph> that need to process most or all of the values from a column. Each data file contains
+      the values for a set of rows (the <q>row group</q>). Within a data file, the values from each column are
+      organized so that they are all adjacent, enabling good compression for the values from that column. Queries
+      against a Parquet table can retrieve and analyze these values from any column quickly and with minimal I/O.
+    </p>
+
+    <table>
+      <title>Parquet Format Support in Impala</title>
+      <tgroup cols="5">
+        <colspec colname="1" colwidth="10*"/>
+        <colspec colname="2" colwidth="10*"/>
+        <colspec colname="3" colwidth="20*"/>
+        <colspec colname="4" colwidth="30*"/>
+        <colspec colname="5" colwidth="30*"/>
+        <thead>
+          <row>
+            <entry>
+              File Type
+            </entry>
+            <entry>
+              Format
+            </entry>
+            <entry>
+              Compression Codecs
+            </entry>
+            <entry>
+              Impala Can CREATE?
+            </entry>
+            <entry>
+              Impala Can INSERT?
+            </entry>
+          </row>
+        </thead>
+        <tbody>
+          <row conref="impala_file_formats.xml#file_formats/parquet_support">
+            <entry/>
+          </row>
+        </tbody>
+      </tgroup>
+    </table>
+
+    <p outputclass="toc inpage"/>
+
+    <p audience="integrated">
+      For general information about using Parquet with other CDH components,
+      see <xref href="cdh_ig_parquet.xml#parquet_format"/>.
+    </p>
+
+  </conbody>
+
+
+  <concept id="parquet_ddl">
+
+    <title>Creating Parquet Tables in Impala</title>
+
+    <conbody>
+
+      <p>
+        To create a table named <codeph>PARQUET_TABLE</codeph> that uses the Parquet format, you would use a
+        command like the following, substituting your own table name, column names, and data types:
+      </p>
+
+<codeblock>[impala-host:21000] &gt; create table <varname>parquet_table_name</varname> (x INT, y STRING) STORED AS PARQUET;</codeblock>
+
+<!--
+<note>
+Formerly, the <codeph>STORED AS</codeph> clause required the keyword <codeph>PARQUETFILE</codeph>.
+In Impala 1.2.2 and higher, you can use <codeph>STORED AS PARQUET</codeph>.
+This <codeph>PARQUET</codeph> keyword is recommended for new code.
+</note>
+-->
+
+      <p>
+        Or, to clone the column names and data types of an existing table:
+      </p>
+
+<codeblock>[impala-host:21000] &gt; create table <varname>parquet_table_name</varname> LIKE <varname>other_table_name</varname> STORED AS PARQUET;</codeblock>
+
+      <p rev="1.4.0">
+        In Impala 1.4.0 and higher, you can derive column definitions from a raw Parquet data file, even without an
+        existing Impala table. For example, you can create an external table pointing to an HDFS directory, and
+        base the column definitions on one of the files in that directory:
+      </p>
+
+<codeblock rev="1.4.0">CREATE EXTERNAL TABLE ingest_existing_files LIKE PARQUET '/user/etl/destination/datafile1.dat'
+  STORED AS PARQUET
+  LOCATION '/user/etl/destination';
+</codeblock>
+
+      <p>
+        Or, you can refer to an existing data file and create a new empty table with suitable column definitions.
+        Then you can use <codeph>INSERT</codeph> to create new data files or <codeph>LOAD DATA</codeph> to transfer
+        existing data files into the new table.
+      </p>
+
+<codeblock rev="1.4.0">CREATE TABLE columns_from_data_file LIKE PARQUET '/user/etl/destination/datafile1.dat'
+  STORED AS PARQUET;
+</codeblock>
+
+      <p>
+        The default properties of the newly created table are the same as for any other <codeph>CREATE
+        TABLE</codeph> statement. For example, the default file format is text; if you want the new table to use
+        the Parquet file format, include the <codeph>STORED AS PARQUET</codeph> file also.
+      </p>
+
+      <p>
+        In this example, the new table is partitioned by year, month, and day. These partition key columns are not
+        part of the data file, so you specify them in the <codeph>CREATE TABLE</codeph> statement:
+      </p>
+
+<codeblock rev="1.4.0">CREATE TABLE columns_from_data_file LIKE PARQUET '/user/etl/destination/datafile1.dat'
+  PARTITION (year INT, month TINYINT, day TINYINT)
+  STORED AS PARQUET;
+</codeblock>
+
+      <p rev="1.4.0">
+        See <xref href="impala_create_table.xml#create_table"/> for more details about the <codeph>CREATE TABLE
+        LIKE PARQUET</codeph> syntax.
+      </p>
+
+      <p>
+        Once you have created a table, to insert data into that table, use a command similar to the following,
+        again with your own table names:
+      </p>
+
+      <!-- To do:
+        Opportunity for another example showing CTAS technique.
+      -->
+
+<codeblock>[impala-host:21000] &gt; insert overwrite table <varname>parquet_table_name</varname> select * from <varname>other_table_name</varname>;</codeblock>
+
+      <p>
+        If the Parquet table has a different number of columns or different column names than the other table,
+        specify the names of columns from the other table rather than <codeph>*</codeph> in the
+        <codeph>SELECT</codeph> statement.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="parquet_etl">
+
+    <title>Loading Data into Parquet Tables</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="ETL"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        Choose from the following techniques for loading data into Parquet tables, depending on whether the
+        original data is already in an Impala table, or exists as raw data files outside Impala.
+      </p>
+
+      <p>
+        If you already have data in an Impala or Hive table, perhaps in a different file format or partitioning
+        scheme, you can transfer the data to a Parquet table using the Impala <codeph>INSERT...SELECT</codeph>
+        syntax. You can convert, filter, repartition, and do other things to the data as part of this same
+        <codeph>INSERT</codeph> statement. See <xref href="#parquet_compression"/> for some examples showing how to
+        insert data into Parquet tables.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/insert_hints"/>
+
+      <p conref="../shared/impala_common.xml#common/insert_parquet_blocksize"/>
+
+      <draft-comment translate="no">
+Add an example here.
+</draft-comment>
+
+      <p>
+        Avoid the <codeph>INSERT...VALUES</codeph> syntax for Parquet tables, because
+        <codeph>INSERT...VALUES</codeph> produces a separate tiny data file for each
+        <codeph>INSERT...VALUES</codeph> statement, and the strength of Parquet is in its handling of data
+        (compressing, parallelizing, and so on) in <ph rev="parquet_block_size">large</ph> chunks.
+      </p>
+
+      <p>
+        If you have one or more Parquet data files produced outside of Impala, you can quickly make the data
+        queryable through Impala by one of the following methods:
+      </p>
+
+      <ul>
+        <li>
+          The <codeph>LOAD DATA</codeph> statement moves a single data file or a directory full of data files into
+          the data directory for an Impala table. It does no validation or conversion of the data. The original
+          data files must be somewhere in HDFS, not the local filesystem.
+          <draft-comment translate="no">
+Add an example here.
+</draft-comment>
+        </li>
+
+        <li>
+          The <codeph>CREATE TABLE</codeph> statement with the <codeph>LOCATION</codeph> clause creates a table
+          where the data continues to reside outside the Impala data directory. The original data files must be
+          somewhere in HDFS, not the local filesystem. For extra safety, if the data is intended to be long-lived
+          and reused by other applications, you can use the <codeph>CREATE EXTERNAL TABLE</codeph> syntax so that
+          the data files are not deleted by an Impala <codeph>DROP TABLE</codeph> statement.
+          <draft-comment translate="no">
+Add an example here.
+</draft-comment>
+        </li>
+
+        <li>
+          If the Parquet table already exists, you can copy Parquet data files directly into it, then use the
+          <codeph>REFRESH</codeph> statement to make Impala recognize the newly added data. Remember to preserve
+          the block size of the Parquet data files by using the <codeph>hadoop distcp -pb</codeph> command rather
+          than a <codeph>-put</codeph> or <codeph>-cp</codeph> operation on the Parquet files. See
+          <xref href="#parquet_compression_multiple"/> for an example of this kind of operation.
+        </li>
+      </ul>
+
+      <note conref="../shared/impala_common.xml#common/restrictions_nonimpala_parquet"/>
+
+      <p>
+        Recent versions of Sqoop can produce Parquet output files using the <codeph>--as-parquetfile</codeph>
+        option.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/sqoop_timestamp_caveat"/>
+
+      <p>
+        If the data exists outside Impala and is in some other format, combine both of the preceding techniques.
+        First, use a <codeph>LOAD DATA</codeph> or <codeph>CREATE EXTERNAL TABLE ... LOCATION</codeph> statement to
+        bring the data into an Impala table that uses the appropriate file format. Then, use an
+        <codeph>INSERT...SELECT</codeph> statement to copy the data to the Parquet table, converting to Parquet
+        format as part of the process.
+      </p>
+
+      <draft-comment translate="no">
+Add an example here.
+</draft-comment>
+
+      <p>
+        Loading data into Parquet tables is a memory-intensive operation, because the incoming data is buffered
+        until it reaches <ph rev="parquet_block_size">one data block</ph> in size, then that chunk of data is
+        organized and compressed in memory before being written out. The memory consumption can be larger when
+        inserting data into partitioned Parquet tables, because a separate data file is written for each
+        combination of partition key column values, potentially requiring several
+        <ph rev="parquet_block_size">large</ph> chunks to be manipulated in memory at once.
+      </p>
+
+      <p>
+        When inserting into a partitioned Parquet table, Impala redistributes the data among the nodes to reduce
+        memory consumption. You might still need to temporarily increase the memory dedicated to Impala during the
+        insert operation, or break up the load operation into several <codeph>INSERT</codeph> statements, or both.
+      </p>
+
+      <note>
+        All the preceding techniques assume that the data you are loading matches the structure of the destination
+        table, including column order, column names, and partition layout. To transform or reorganize the data,
+        start by loading the data into a Parquet table that matches the underlying structure of the data, then use
+        one of the table-copying techniques such as <codeph>CREATE TABLE AS SELECT</codeph> or <codeph>INSERT ...
+        SELECT</codeph> to reorder or rename columns, divide the data among multiple partitions, and so on. For
+        example to take a single comprehensive Parquet data file and load it into a partitioned table, you would
+        use an <codeph>INSERT ... SELECT</codeph> statement with dynamic partitioning to let Impala create separate
+        data files with the appropriate partition values; for an example, see
+        <xref href="impala_insert.xml#insert"/>.
+      </note>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="parquet_performance">
+
+    <title>Query Performance for Impala Parquet Tables</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Performance"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        Query performance for Parquet tables depends on the number of columns needed to process the
+        <codeph>SELECT</codeph> list and <codeph>WHERE</codeph> clauses of the query, the way data is divided into
+        <ph rev="parquet_block_size">large data files with block size equal to file size</ph>, the reduction in I/O
+        by reading the data for each column in compressed format, which data files can be skipped (for partitioned
+        tables), and the CPU overhead of decompressing the data for each column.
+      </p>
+
+      <p>
+        For example, the following is an efficient query for a Parquet table:
+<codeblock>select avg(income) from census_data where state = 'CA';</codeblock>
+        The query processes only 2 columns out of a large number of total columns. If the table is partitioned by
+        the <codeph>STATE</codeph> column, it is even more efficient because the query only has to read and decode
+        1 column from each data file, and it can read only the data files in the partition directory for the state
+        <codeph>'CA'</codeph>, skipping the data files for all the other states, which will be physically located
+        in other directories.
+      </p>
+
+      <p>
+        The following is a relatively inefficient query for a Parquet table:
+<codeblock>select * from census_data;</codeblock>
+        Impala would have to read the entire contents of each <ph rev="parquet_block_size">large</ph> data file,
+        and decompress the contents of each column for each row group, negating the I/O optimizations of the
+        column-oriented format. This query might still be faster for a Parquet table than a table with some other
+        file format, but it does not take advantage of the unique strengths of Parquet data files.
+      </p>
+
+      <p>
+        Impala can optimize queries on Parquet tables, especially join queries, better when statistics are
+        available for all the tables. Issue the <codeph>COMPUTE STATS</codeph> statement for each table after
+        substantial amounts of data are loaded into or appended to it. See
+        <xref href="impala_compute_stats.xml#compute_stats"/> for details.
+      </p>
+
+      <p rev="2.5.0">
+        The runtime filtering feature, available in <keyword keyref="impala25_full"/> and higher, works best with Parquet tables.
+        The per-row filtering aspect only applies to Parquet tables.
+        See <xref href="impala_runtime_filtering.xml#runtime_filtering"/> for details.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/s3_block_splitting"/>
+
+    </conbody>
+
+    <concept id="parquet_partitioning">
+
+      <title>Partitioning for Parquet Tables</title>
+
+      <conbody>
+
+        <p>
+          As explained in <xref href="impala_partitioning.xml#partitioning"/>, partitioning is an important
+          performance technique for Impala generally. This section explains some of the performance considerations
+          for partitioned Parquet tables.
+        </p>
+
+        <p>
+          The Parquet file format is ideal for tables containing many columns, where most queries only refer to a
+          small subset of the columns. As explained in <xref href="#parquet_data_files"/>, the physical layout of
+          Parquet data files lets Impala read only a small fraction of the data for many queries. The performance
+          benefits of this approach are amplified when you use Parquet tables in combination with partitioning.
+          Impala can skip the data files for certain partitions entirely, based on the comparisons in the
+          <codeph>WHERE</codeph> clause that refer to the partition key columns. For example, queries on
+          partitioned tables often analyze data for time intervals based on columns such as <codeph>YEAR</codeph>,
+          <codeph>MONTH</codeph>, and/or <codeph>DAY</codeph>, or for geographic regions. Remember that Parquet
+          data files use a <ph rev="parquet_block_size">large</ph> block size, so when deciding how finely to
+          partition the data, try to find a granularity where each partition contains
+          <ph rev="parquet_block_size">256 MB</ph> or more of data, rather than creating a large number of smaller
+          files split among many partitions.
+        </p>
+
+        <p>
+          Inserting into a partitioned Parquet table can be a resource-intensive operation, because each Impala
+          node could potentially be writing a separate data file to HDFS for each combination of different values
+          for the partition key columns. The large number of simultaneous open files could exceed the HDFS
+          <q>transceivers</q> limit. To avoid exceeding this limit, consider the following techniques:
+        </p>
+
+        <ul>
+          <li>
+            Load different subsets of data using separate <codeph>INSERT</codeph> statements with specific values
+            for the <codeph>PARTITION</codeph> clause, such as <codeph>PARTITION (year=2010)</codeph>.
+          </li>
+
+          <li>
+            Increase the <q>transceivers</q> value for HDFS, sometimes spelled <q>xcievers</q> (sic). The property
+            value in the <filepath>hdfs-site.xml</filepath> configuration file is
+<!-- Old name, now deprecated: <codeph>dfs.datanode.max.xcievers</codeph>. -->
+            <codeph>dfs.datanode.max.transfer.threads</codeph>. For example, if you were loading 12 years of data
+            partitioned by year, month, and day, even a value of 4096 might not be high enough. This
+            <xref href="http://blog.cloudera.com/blog/2012/03/hbase-hadoop-xceivers/" scope="external" format="html">blog
+            post</xref> explores the considerations for setting this value higher or lower, using HBase examples
+            for illustration.
+          </li>
+
+          <li>
+            Use the <codeph>COMPUTE STATS</codeph> statement to collect
+            <xref href="impala_perf_stats.xml#perf_column_stats">column statistics</xref> on the source table from
+            which data is being copied, so that the Impala query can estimate the number of different values in the
+            partition key columns and distribute the work accordingly.
+          </li>
+        </ul>
+
+      </conbody>
+
+    </concept>
+
+  </concept>
+
+  <concept id="parquet_compression">
+
+    <title>Snappy and GZip Compression for Parquet Data Files</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Snappy"/>
+      <data name="Category" value="Gzip"/>
+      <data name="Category" value="Compression"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">COMPRESSION_CODEC query option</indexterm>
+        When Impala writes Parquet data files using the <codeph>INSERT</codeph> statement, the underlying
+        compression is controlled by the <codeph>COMPRESSION_CODEC</codeph> query option. (Prior to Impala 2.0, the
+        query option name was <codeph>PARQUET_COMPRESSION_CODEC</codeph>.) The allowed values for this query option
+        are <codeph>snappy</codeph> (the default), <codeph>gzip</codeph>, and <codeph>none</codeph>. The option
+        value is not case-sensitive. If the option is set to an unrecognized value, all kinds of queries will fail
+        due to the invalid option setting, not just queries involving Parquet tables.
+      </p>
+
+    </conbody>
+
+    <concept id="parquet_snappy">
+
+      <title>Example of Parquet Table with Snappy Compression</title>
+
+      <conbody>
+
+        <p>
+          <indexterm audience="Cloudera">compression</indexterm>
+          By default, the underlying data files for a Parquet table are compressed with Snappy. The combination of
+          fast compression and decompression makes it a good choice for many data sets. To ensure Snappy
+          compression is used, for example after experimenting with other compression codecs, set the
+          <codeph>COMPRESSION_CODEC</codeph> query option to <codeph>snappy</codeph> before inserting the data:
+        </p>
+
+<codeblock>[localhost:21000] &gt; create database parquet_compression;
+[localhost:21000] &gt; use parquet_compression;
+[localhost:21000] &gt; create table parquet_snappy like raw_text_data;
+[localhost:21000] &gt; set COMPRESSION_CODEC=snappy;
+[localhost:21000] &gt; insert into parquet_snappy select * from raw_text_data;
+Inserted 1000000000 rows in 181.98s
+</codeblock>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="parquet_gzip">
+
+      <title>Example of Parquet Table with GZip Compression</title>
+
+      <conbody>
+
+        <p>
+          If you need more intensive compression (at the expense of more CPU cycles for uncompressing during
+          queries), set the <codeph>COMPRESSION_CODEC</codeph> query option to <codeph>gzip</codeph> before
+          inserting the data:
+        </p>
+
+<codeblock>[localhost:21000] &gt; create table parquet_gzip like raw_text_data;
+[localhost:21000] &gt; set COMPRESSION_CODEC=gzip;
+[localhost:21000] &gt; insert into parquet_gzip select * from raw_text_data;
+Inserted 1000000000 rows in 1418.24s
+</codeblock>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="parquet_none">
+
+      <title>Example of Uncompressed Parquet Table</title>
+
+      <conbody>
+
+        <p>
+          If your data compresses very poorly, or you want to avoid the CPU overhead of compression and
+          decompression entirely, set the <codeph>COMPRESSION_CODEC</codeph> query option to <codeph>none</codeph>
+          before inserting the data:
+        </p>
+
+<codeblock>[localhost:21000] &gt; create table parquet_none like raw_text_data;
+[localhost:21000] &gt; set COMPRESSION_CODEC=none;
+[localhost:21000] &gt; insert into parquet_none select * from raw_text_data;
+Inserted 1000000000 rows in 146.90s
+</codeblock>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="parquet_compression_examples">
+
+      <title>Examples of Sizes and Speeds for Compressed Parquet Tables</title>
+
+      <conbody>
+
+        <p>
+          Here are some examples showing differences in data sizes and query speeds for 1 billion rows of synthetic
+          data, compressed with each kind of codec. As always, run similar tests with realistic data sets of your
+          own. The actual compression ratios, and relative insert and query speeds, will vary depending on the
+          characteristics of the actual data.
+        </p>
+
+        <p>
+          In this case, switching from Snappy to GZip compression shrinks the data by an additional 40% or so,
+          while switching from Snappy compression to no compression expands the data also by about 40%:
+        </p>
+
+<codeblock>$ hdfs dfs -du -h /user/hive/warehouse/parquet_compression.db
+23.1 G  /user/hive/warehouse/parquet_compression.db/parquet_snappy
+13.5 G  /user/hive/warehouse/parquet_compression.db/parquet_gzip
+32.8 G  /user/hive/warehouse/parquet_compression.db/parquet_none
+</codeblock>
+
+        <p>
+          Because Parquet data files are typically <ph rev="parquet_block_size">large</ph>, each directory will
+          have a different number of data files and the row groups will be arranged differently.
+        </p>
+
+        <p>
+          At the same time, the less agressive the compression, the faster the data can be decompressed. In this
+          case using a table with a billion rows, a query that evaluates all the values for a particular column
+          runs faster with no compression than with Snappy compression, and faster with Snappy compression than
+          with Gzip compression. Query performance depends on several other factors, so as always, run your own
+          benchmarks with your own data to determine the ideal tradeoff between data size, CPU efficiency, and
+          speed of insert and query operations.
+        </p>
+
+<codeblock>[localhost:21000] &gt; desc parquet_snappy;
+Query finished, fetching results ...
++-----------+---------+---------+
+| name      | type    | comment |
++-----------+---------+---------+
+| id        | int     |         |
+| val       | int     |         |
+| zfill     | string  |         |
+| name      | string  |         |
+| assertion | boolean |         |
++-----------+---------+---------+
+Returned 5 row(s) in 0.14s
+[localhost:21000] &gt; select avg(val) from parquet_snappy;
+Query finished, fetching results ...
++-----------------+
+| _c0             |
++-----------------+
+| 250000.93577915 |
++-----------------+
+Returned 1 row(s) in 4.29s
+[localhost:21000] &gt; select avg(val) from parquet_gzip;
+Query finished, fetching results ...
++-----------------+
+| _c0             |
++-----------------+
+| 250000.93577915 |
++-----------------+
+Returned 1 row(s) in 6.97s
+[localhost:21000] &gt; select avg(val) from parquet_none;
+Query finished, fetching results ...
++-----------------+
+| _c0             |
++-----------------+
+| 250000.93577915 |
++-----------------+
+Returned 1 row(s) in 3.67s
+</codeblock>
+
+      </conbody>
+
+    </concept>
+
+    <concept id="parquet_compression_multiple">
+
+      <title>Example of Copying Parquet Data Files</title>
+
+      <conbody>
+
+        <p>
+          Here is a final example, to illustrate how the data files using the various compression codecs are all
+          compatible with each other for read operations. The metadata about the compression format is written into
+          each data file, and can be decoded during queries regardless of the <codeph>COMPRESSION_CODEC</codeph>
+          setting in effect at the time. In this example, we copy data files from the
+          <codeph>PARQUET_SNAPPY</codeph>, <codeph>PARQUET_GZIP</codeph>, and <codeph>PARQUET_NONE</codeph> tables
+          used in the previous examples, each containing 1 billion rows, all to the data directory of a new table
+          <codeph>PARQUET_EVERYTHING</codeph>. A couple of sample queries demonstrate that the new table now
+          contains 3 billion rows featuring a variety of compression codecs for the data files.
+        </p>
+
+        <p>
+          First, we create the table in Impala so that there is a destination directory in HDFS to put the data
+          files:
+        </p>
+
+<codeblock>[localhost:21000] &gt; create table parquet_everything like parquet_snappy;
+Query: create table parquet_everything like parquet_snappy
+</codeblock>
+
+        <p>
+          Then in the shell, we copy the relevant data files into the data directory for this new table. Rather
+          than using <codeph>hdfs dfs -cp</codeph> as with typical files, we use <codeph>hadoop distcp -pb</codeph>
+          to ensure that the special <ph rev="parquet_block_size"> block size</ph> of the Parquet data files is
+          preserved.
+        </p>
+
+<codeblock>$ hadoop distcp -pb /user/hive/warehouse/parquet_compression.db/parquet_snappy \
+  /user/hive/warehouse/parquet_compression.db/parquet_everything
+...<varname>MapReduce output</varname>...
+$ hadoop distcp -pb /user/hive/warehouse/parquet_compression.db/parquet_gzip  \
+  /user/hive/warehouse/parquet_compression.db/parquet_everything
+...<varname>MapReduce output</varname>...
+$ hadoop distcp -pb /user/hive/warehouse/parquet_compression.db/parquet_none  \
+  /user/hive/warehouse/parquet_compression.db/parquet_everything
+...<varname>MapReduce output</varname>...
+</codeblock>
+
+        <p>
+          Back in the <cmdname>impala-shell</cmdname> interpreter, we use the <codeph>REFRESH</codeph> statement to
+          alert the Impala server to the new data files for this table, then we can run queries demonstrating that
+          the data files represent 3 billion rows, and the values for one of the numeric columns match what was in
+          the original smaller tables:
+        </p>
+
+<codeblock>[localhost:21000] &gt; refresh parquet_everything;
+Query finished, fetching results ...
+
+Returned 0 row(s) in 0.32s
+[localhost:21000] &gt; select count(*) from parquet_everything;
+Query finished, fetching results ...
++------------+
+| _c0        |
++------------+
+| 3000000000 |
++------------+
+Returned 1 row(s) in 8.18s
+[localhost:21000] &gt; select avg(val) from parquet_everything;
+Query finished, fetching results ...
++-----------------+
+| _c0             |
++-----------------+
+| 250000.93577915 |
++-----------------+
+Returned 1 row(s) in 13.35s
+</codeblock>
+
+      </conbody>
+
+    </concept>
+
+  </concept>
+
+  <concept rev="2.3.0" id="parquet_complex_types">
+
+    <title>Parquet Tables for Impala Complex Types</title>
+
+    <conbody>
+
+    <p>
+      In <keyword keyref="impala23_full"/> and higher, Impala supports the complex types
+      <codeph>ARRAY</codeph>, <codeph>STRUCT</codeph>, and <codeph>MAP</codeph>
+      See <xref href="impala_complex_types.xml#complex_types"/> for details.
+      Because these data types are currently supported only for the Parquet file format,
+      if you plan to use them, become familiar with the performance and storage aspects
+      of Parquet first.
+    </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="parquet_interop">
+
+    <title>Exchanging Parquet Data Files with Other Hadoop Components</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Hadoop"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        Starting in CDH 4.5, you can read and write Parquet data files from other CDH components.
+        <ph audience="integrated">See <xref href="cdh_ig_parquet.xml#parquet_format"/> for details.</ph>
+      </p>
+
+<!-- These couple of paragraphs reused in the release notes 'incompatible changes' section. -->
+
+<!-- But conbodydiv tag too restrictive, can't have just paragraphs and codeblocks inside. -->
+
+<!-- So I will physically copy the info for the time being. -->
+
+<!-- <conbodydiv id="upgrade_parquet_metadata"> -->
+
+      <p>
+        Previously, it was not possible to create Parquet data through Impala and reuse that table within Hive. Now
+        that Parquet support is available for Hive in CDH 4.5, reusing existing Impala Parquet data files in Hive
+        requires updating the table metadata. Use the following command if you are already running Impala 1.1.1 or
+        higher:
+      </p>
+
+<codeblock>ALTER TABLE <varname>table_name</varname> SET FILEFORMAT PARQUET;
+</codeblock>
+
+      <p>
+        If you are running a level of Impala that is older than 1.1.1, do the metadata update through Hive:
+      </p>
+
+<codeblock>ALTER TABLE <varname>table_name</varname> SET SERDE 'parquet.hive.serde.ParquetHiveSerDe';
+ALTER TABLE <varname>table_name</varname> SET FILEFORMAT
+  INPUTFORMAT "parquet.hive.DeprecatedParquetInputFormat"
+  OUTPUTFORMAT "parquet.hive.DeprecatedParquetOutputFormat";
+</codeblock>
+
+      <p>
+        Impala 1.1.1 and higher can reuse Parquet data files created by Hive, without any action required.
+      </p>
+
+<!-- </conbodydiv> -->
+
+      <p rev="2.2.0">
+        Impala supports the scalar data types that you can encode in a Parquet data file, but not composite or
+        nested types such as maps or arrays. In <keyword keyref="impala22_full"/> and higher, Impala can query Parquet data
+        files that include composite or nested types, as long as the query only refers to columns with scalar
+        types.
+<!-- TK: could include an example here, but would require setup in Hive or Pig or something. -->
+      </p>
+
+      <p>
+        If you copy Parquet data files between nodes, or even between different directories on the same node, make
+        sure to preserve the block size by using the command <codeph>hadoop distcp -pb</codeph>. To verify that the
+        block size was preserved, issue the command <codeph>hdfs fsck -blocks
+        <varname>HDFS_path_of_impala_table_dir</varname></codeph> and check that the average block size is at or
+        near <ph rev="parquet_block_size">256 MB (or whatever other size is defined by the
+        <codeph>PARQUET_FILE_SIZE</codeph> query option).</ph>. (The <codeph>hadoop distcp</codeph> operation
+        typically leaves some directories behind, with names matching <filepath>_distcp_logs_*</filepath>, that you
+        can delete from the destination directory afterward.)
+<!-- The Apache wiki page keeps disappearing, even though Google still points to it as of Nov. 11/2014. -->
+<!-- Now there is a 'distcp2' guide: http://hadoop.apache.org/docs/r1.2.1/distcp2.html but I haven't tried that so let's play it safe for now and hide the link. -->
+<!--      See the <xref href="http://hadoop.apache.org/docs/r0.19.0/distcp.html" scope="external" format="html">Hadoop DistCP Guide</xref> for details. -->
+        Issue the command <cmdname>hadoop distcp</cmdname> for details about <cmdname>distcp</cmdname> command
+        syntax.
+      </p>
+
+<!-- Sample commands/output for when the 'distcp' business is expanded into a tutorial later. Part of
+     a1730.halxg.cloudera.com:/home/jrussell/jdr/mixed_format_partitions.log.
+<codeblock>$ hdfs fsck -blocks /user/impala/warehouse/parquet_compression.db/parquet_everything
+Connecting to namenode via http://a1730.halxg.cloudera.com:50070
+FSCK started by jrussell (auth:SIMPLE) from /10.20.198.130 for path /user/impala/warehouse/parquet_compression.db/parquet_everything at Fri Aug 23 11:35:37 PDT 2013
+............................................................................Status: HEALTHY
+ Total size:    74504481213 B
+ Total dirs:    1
+ Total files:   76
+ Total blocks (validated):      76 (avg. block size 980322121 B)
+ Minimally replicated blocks:   76 (100.0 %)
+ Over-replicated blocks:        0 (0.0 %)
+ Under-replicated blocks:       0 (0.0 %)
+ Mis-replicated blocks:         0 (0.0 %)
+ Default replication factor:    3
+ Average block replication:     3.0
+ Corrupt blocks:                0
+ Missing replicas:              0 (0.0 %)
+ Number of data-nodes:          4
+ Number of racks:               1
+FSCK ended at Fri Aug 23 11:35:37 PDT 2013 in 8 milliseconds
+
+
+The filesystem under path '/user/impala/warehouse/parquet_compression.db/parquet_everything' is HEALTHY
+</codeblock>
+-->
+
+      <p conref="../shared/impala_common.xml#common/impala_parquet_encodings_caveat"/>
+      <p conref="../shared/impala_common.xml#common/parquet_tools_blurb"/>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="parquet_data_files">
+
+    <title>How Parquet Data Files Are Organized</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Concepts"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        Although Parquet is a column-oriented file format, do not expect to find one data file for each column.
+        Parquet keeps all the data for a row within the same data file, to ensure that the columns for a row are
+        always available on the same node for processing. What Parquet does is to set a large HDFS block size and a
+        matching maximum data file size, to ensure that I/O and network transfer requests apply to large batches of
+        data.
+      </p>
+
+      <p>
+        Within that data file, the data for a set of rows is rearranged so that all the values from the first
+        column are organized in one contiguous block, then all the values from the second column, and so on.
+        Putting the values from the same column next to each other lets Impala use effective compression techniques
+        on the values in that column.
+      </p>
+
+      <note>
+        <p>
+          Impala <codeph>INSERT</codeph> statements write Parquet data files using an HDFS block size
+          <ph rev="parquet_block_size">that matches the data file size</ph>, to ensure that each data file is
+          represented by a single HDFS block, and the entire file can be processed on a single node without
+          requiring any remote reads.
+        </p>
+
+        <p>
+          If you create Parquet data files outside of Impala, such as through a MapReduce or Pig job, ensure that
+          the HDFS block size is greater than or equal to the file size, so that the <q>one file per block</q>
+          relationship is maintained. Set the <codeph>dfs.block.size</codeph> or the <codeph>dfs.blocksize</codeph>
+          property large enough that each file fits within a single HDFS block, even if that size is larger than
+          the normal HDFS block size.
+        </p>
+
+        <p>
+          If the block size is reset to a lower value during a file copy, you will see lower performance for
+          queries involving those files, and the <codeph>PROFILE</codeph> statement will reveal that some I/O is
+          being done suboptimally, through remote reads. See
+          <xref href="impala_parquet.xml#parquet_compression_multiple"/> for an example showing how to preserve the
+          block size when copying Parquet data files.
+        </p>
+      </note>
+
+      <p>
+        When Impala retrieves or tests the data for a particular column, it opens all the data files, but only
+        reads the portion of each file containing the values for that column. The column values are stored
+        consecutively, minimizing the I/O required to process the values within a single column. If other columns
+        are named in the <codeph>SELECT</codeph> list or <codeph>WHERE</codeph> clauses, the data for all columns
+        in the same row is available within that same data file.
+      </p>
+
+      <p>
+        If an <codeph>INSERT</codeph> statement brings in less than <ph rev="parquet_block_size">one Parquet
+        block's worth</ph> of data, the resulting data file is smaller than ideal. Thus, if you do split up an ETL
+        job to use multiple <codeph>INSERT</codeph> statements, try to keep the volume of data for each
+        <codeph>INSERT</codeph> statement to approximately <ph rev="parquet_block_size">256 MB, or a multiple of
+        256 MB</ph>.
+      </p>
+
+    </conbody>
+
+    <concept id="parquet_encoding">
+
+      <title>RLE and Dictionary Encoding for Parquet Data Files</title>
+
+      <conbody>
+
+        <p>
+          Parquet uses some automatic compression techniques, such as run-length encoding (RLE) and dictionary
+          encoding, based on analysis of the actual data values. Once the data values are encoded in a compact
+          form, the encoded data can optionally be further compressed using a compression algorithm. Parquet data
+          files created by Impala can use Snappy, GZip, or no compression; the Parquet spec also allows LZO
+          compression, but currently Impala does not support LZO-compressed Parquet files.
+        </p>
+
+        <p>
+          RLE and dictionary encoding are compression techniques that Impala applies automatically to groups of
+          Parquet data values, in addition to any Snappy or GZip compression applied to the entire data files.
+          These automatic optimizations can save you time and planning that are normally needed for a traditional
+          data warehouse. For example, dictionary encoding reduces the need to create numeric IDs as abbreviations
+          for longer string values.
+        </p>
+
+        <p>
+          Run-length encoding condenses sequences of repeated data values. For example, if many consecutive rows
+          all contain the same value for a country code, those repeating values can be represented by the value
+          followed by a count of how many times it appears consecutively.
+        </p>
+
+        <p>
+          Dictionary encoding takes the different values present in a column, and represents each one in compact
+          2-byte form rather than the original value, which could be several bytes. (Additional compression is
+          applied to the compacted values, for extra space savings.) This type of encoding applies when the number
+          of different values for a column is less than 2**16 (16,384). It does not apply to columns of data type
+          <codeph>BOOLEAN</codeph>, which are already very short. <codeph>TIMESTAMP</codeph> columns sometimes have
+          a unique value for each row, in which case they can quickly exceed the 2**16 limit on distinct values.
+          The 2**16 limit on different values within a column is reset for each data file, so if several different
+          data files each contained 10,000 different city names, the city name column in each data file could still
+          be condensed using dictionary encoding.
+        </p>
+
+      </conbody>
+
+    </concept>
+
+  </concept>
+
+  <concept rev="1.4.0" id="parquet_compacting">
+
+    <title>Compacting Data Files for Parquet Tables</title>
+
+    <conbody>
+
+      <p>
+        If you reuse existing table structures or ETL processes for Parquet tables, you might encounter a <q>many
+        small files</q> situation, which is suboptimal for query efficiency. For example, statements like these
+        might produce inefficiently organized data files:
+      </p>
+
+<codeblock>-- In an N-node cluster, each node produces a data file
+-- for the INSERT operation. If you have less than
+-- N GB of data to copy, some files are likely to be
+-- much smaller than the <ph rev="parquet_block_size">default Parquet</ph> block size.
+insert into parquet_table select * from text_table;
+
+-- Even if this operation involves an overall large amount of data,
+-- when split up by year/month/day, each partition might only
+-- receive a small amount of data. Then the data files for
+-- the partition might be divided between the N nodes in the cluster.
+-- A multi-gigabyte copy operation might produce files of only
+-- a few MB each.
+insert into partitioned_parquet_table partition (year, month, day)
+  select year, month, day, url, referer, user_agent, http_code, response_time
+  from web_stats;
+</codeblock>
+
+      <p>
+        Here are techniques to help you produce large data files in Parquet <codeph>INSERT</codeph> operations, and
+        to compact existing too-small data files:
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            When inserting into a partitioned Parquet table, use statically partitioned <codeph>INSERT</codeph>
+            statements where the partition key values are specified as constant values. Ideally, use a separate
+            <codeph>INSERT</codeph> statement for each partition.
+          </p>
+        </li>
+
+        <li>
+          <p conref="../shared/impala_common.xml#common/num_nodes_tip"/>
+        </li>
+
+        <li>
+          <p>
+            Be prepared to reduce the number of partition key columns from what you are used to with traditional
+            analytic database systems.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Do not expect Impala-written Parquet files to fill up the entire Parquet block size. Impala estimates
+            on the conservative side when figuring out how much data to write to each Parquet file. Typically, the
+            of uncompressed data in memory is substantially reduced on disk by the compression and encoding
+            techniques in the Parquet file format.
+<!--
+  Impala reserves <ph rev="parquet_block_size">1 GB</ph> of memory to buffer the data before writing,
+  but the actual data file might be smaller, in the hundreds of megabytes.
+  -->
+            The final data file size varies depending on the compressibility of the data. Therefore, it is not an
+            indication of a problem if <ph rev="parquet_block_size">256 MB</ph> of text data is turned into 2
+            Parquet data files, each less than <ph rev="parquet_block_size">256 MB</ph>.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            If you accidentally end up with a table with many small data files, consider using one or more of the
+            preceding techniques and copying all the data into a new Parquet table, either through <codeph>CREATE
+            TABLE AS SELECT</codeph> or <codeph>INSERT ... SELECT</codeph> statements.
+          </p>
+
+          <p>
+            To avoid rewriting queries to change table names, you can adopt a convention of always running
+            important queries against a view. Changing the view definition immediately switches any subsequent
+            queries to use the new underlying tables:
+          </p>
+<codeblock>create view production_table as select * from table_with_many_small_files;
+-- CTAS or INSERT...SELECT all the data into a more efficient layout...
+alter view production_table as select * from table_with_few_big_files;
+select * from production_table where c1 = 100 and c2 &lt; 50 and ...;
+</codeblock>
+        </li>
+      </ul>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="1.4.0" id="parquet_schema_evolution">
+
+    <title>Schema Evolution for Parquet Tables</title>
+
+    <conbody>
+
+      <p>
+        Schema evolution refers to using the statement <codeph>ALTER TABLE ... REPLACE COLUMNS</codeph> to change
+        the names, data type, or number of columns in a table. You can perform schema evolution for Parquet tables
+        as follows:
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            The Impala <codeph>ALTER TABLE</codeph> statement never changes any data files in the tables. From the
+            Impala side, schema evolution involves interpreting the same data files in terms of a new table
+            definition. Some types of schema changes make sense and are represented correctly. Other types of
+            changes cannot be represented in a sensible way, and produce special result values or conversion errors
+            during queries.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            The <codeph>INSERT</codeph> statement always creates data using the latest table definition. You might
+            end up with data files with different numbers of columns or internal data representations if you do a
+            sequence of <codeph>INSERT</codeph> and <codeph>ALTER TABLE ... REPLACE COLUMNS</codeph> statements.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            If you use <codeph>ALTER TABLE ... REPLACE COLUMNS</codeph> to define additional columns at the end,
+            when the original data files are used in a query, these final columns are considered to be all
+            <codeph>NULL</codeph> values.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            If you use <codeph>ALTER TABLE ... REPLACE COLUMNS</codeph> to define fewer columns than before, when
+            the original data files are used in a query, the unused columns still present in the data file are
+            ignored.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Parquet represents the <codeph>TINYINT</codeph>, <codeph>SMALLINT</codeph>, and <codeph>INT</codeph>
+            types the same internally, all stored in 32-bit integers.
+          </p>
+          <ul>
+            <li>
+              That means it is easy to promote a <codeph>TINYINT</codeph> column to <codeph>SMALLINT</codeph> or
+              <codeph>INT</codeph>, or a <codeph>SMALLINT</codeph> column to <codeph>INT</codeph>. The numbers are
+              represented exactly the same in the data file, and the columns being promoted would not contain any
+              out-of-range values.
+            </li>
+
+            <li>
+              <p>
+                If you change any of these column types to a smaller type, any values that are out-of-range for the
+                new type are returned incorrectly, typically as negative numbers.
+              </p>
+            </li>
+
+            <li>
+              <p>
+                You cannot change a <codeph>TINYINT</codeph>, <codeph>SMALLINT</codeph>, or <codeph>INT</codeph>
+                column to <codeph>BIGINT</codeph>, or the other way around. Although the <codeph>ALTER
+                TABLE</codeph> succeeds, any attempt to query those columns results in conversion errors.
+              </p>
+            </li>
+
+            <li>
+              <p>
+                Any other type conversion for columns produces a conversion error during queries. For example,
+                <codeph>INT</codeph> to <codeph>STRING</codeph>, <codeph>FLOAT</codeph> to <codeph>DOUBLE</codeph>,
+                <codeph>TIMESTAMP</codeph> to <codeph>STRING</codeph>, <codeph>DECIMAL(9,0)</codeph> to
+                <codeph>DECIMAL(5,2)</codeph>, and so on.
+              </p>
+            </li>
+          </ul>
+        </li>
+      </ul>
+
+      <p rev="2.6.0 IMPALA-2835 CDH-33330">
+        You might find that you have Parquet files where the columns do not line up in the same
+        order as in your Impala table. For example, you might have a Parquet file that was part of
+        a table with columns <codeph>C1,C2,C3,C4</codeph>, and now you want to reuse the same
+        Parquet file in a table with columns <codeph>C4,C2</codeph>. By default, Impala expects the
+        columns in the data file to appear in the same order as the columns defined for the table,
+        making it impractical to do some kinds of file reuse or schema evolution. In <keyword keyref="impala26_full"/>
+        and higher, the query option <codeph>PARQUET_FALLBACK_SCHEMA_RESOLUTION=name</codeph> lets Impala
+        resolve columns by name, and therefore handle out-of-order or extra columns in the data file.
+        For example:
+
+<codeblock conref="../shared/impala_common.xml#common/parquet_fallback_schema_resolution_example"/>
+
+        See <xref href="impala_parquet_fallback_schema_resolution.xml#parquet_fallback_schema_resolution"/>
+        for more details.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+  <concept id="parquet_data_types">
+
+    <title>Data Type Considerations for Parquet Tables</title>
+
+    <conbody>
+
+      <p>
+        The Parquet format defines a set of data types whose names differ from the names of the corresponding
+        Impala data types. If you are preparing Parquet files using other Hadoop components such as Pig or
+        MapReduce, you might need to work with the type names defined by Parquet. The following figure lists the
+        Parquet-defined types and the equivalent types in Impala.
+      </p>
+
+      <p>
+        <b>Primitive types:</b>
+      </p>
+
+<codeblock>BINARY -&gt; STRING
+BOOLEAN -&gt; BOOLEAN
+DOUBLE -&gt; DOUBLE
+FLOAT -&gt; FLOAT
+INT32 -&gt; INT
+INT64 -&gt; BIGINT
+INT96 -&gt; TIMESTAMP
+</codeblock>
+
+      <p>
+        <b>Logical types:</b>
+      </p>
+
+<codeblock>BINARY + OriginalType UTF8 -&gt; STRING
+BINARY + OriginalType DECIMAL -&gt; DECIMAL
+</codeblock>
+
+      <p rev="2.3.0">
+        <b>Complex types:</b>
+      </p>
+
+      <p rev="2.3.0">
+        For the complex types (<codeph>ARRAY</codeph>, <codeph>MAP</codeph>, and <codeph>STRUCT</codeph>)
+        available in <keyword keyref="impala23_full"/> and higher, Impala only supports queries
+        against those types in Parquet tables.
+      </p>
+
+    </conbody>
+
+  </concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_parquet_annotate_strings_utf8.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_parquet_annotate_strings_utf8.xml b/docs/topics/impala_parquet_annotate_strings_utf8.xml
new file mode 100644
index 0000000..b603b2c
--- /dev/null
+++ b/docs/topics/impala_parquet_annotate_strings_utf8.xml
@@ -0,0 +1,50 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="parquet_annotate_strings_utf8" rev="2.6.0 IMPALA-2069">
+
+  <title>PARQUET_ANNOTATE_STRINGS_UTF8 Query Option (<keyword keyref="impala26"/> or higher only)</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Parquet"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.6.0 IMPALA-2069">
+      <indexterm audience="Cloudera">PARQUET_ANNOTATE_STRINGS_UTF8 query option</indexterm>
+      Causes Impala <codeph>INSERT</codeph> and <codeph>CREATE TABLE AS SELECT</codeph> statements
+      to write Parquet files that use the UTF-8 annotation for <codeph>STRING</codeph> columns.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+    <p>
+      By default, Impala represents a <codeph>STRING</codeph> column in Parquet as an unannotated binary field.
+    </p>
+    <p>
+      Impala always uses the UTF-8 annotation when writing <codeph>CHAR</codeph> and <codeph>VARCHAR</codeph>
+      columns to Parquet files. An alternative to using the query option is to cast <codeph>STRING</codeph>
+      values to <codeph>VARCHAR</codeph>.
+    </p>
+    <p>
+      This option is to help make Impala-written data more interoperable with other data processing engines.
+      Impala itself currently does not support all operations on UTF-8 data.
+      Although data processed by Impala is typically represented in ASCII, it is valid to designate the
+      data as UTF-8 when storing on disk, because ASCII is a subset of UTF-8.
+    </p>
+    <p conref="../shared/impala_common.xml#common/type_boolean"/>
+    <p conref="../shared/impala_common.xml#common/default_false_0"/>
+
+    <p conref="../shared/impala_common.xml#common/added_in_260"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_parquet.xml#parquet"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_parquet_compression_codec.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_parquet_compression_codec.xml b/docs/topics/impala_parquet_compression_codec.xml
new file mode 100644
index 0000000..7132727
--- /dev/null
+++ b/docs/topics/impala_parquet_compression_codec.xml
@@ -0,0 +1,26 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="parquet_compression_codec">
+
+  <title>PARQUET_COMPRESSION_CODEC Query Option</title>
+  <titlealts audience="PDF"><navtitle>PARQUET_COMPRESSION_CODEC</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Parquet"/>
+      <data name="Category" value="File Formats"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Deprecated Features"/>
+      <data name="Category" value="Compression"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">PARQUET_COMPRESSION_CODEC query option</indexterm>
+      Deprecated. Use <codeph>COMPRESSION_CODEC</codeph> in Impala 2.0 and later. See
+      <xref href="impala_compression_codec.xml#compression_codec"/> for details.
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_parquet_fallback_schema_resolution.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_parquet_fallback_schema_resolution.xml b/docs/topics/impala_parquet_fallback_schema_resolution.xml
new file mode 100644
index 0000000..e9c9327
--- /dev/null
+++ b/docs/topics/impala_parquet_fallback_schema_resolution.xml
@@ -0,0 +1,49 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="parquet_fallback_schema_resolution" rev="2.6.0 IMPALA-2835 CDH-33330">
+
+  <title>PARQUET_FALLBACK_SCHEMA_RESOLUTION Query Option (<keyword keyref="impala26"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>PARQUET_FALLBACK_SCHEMA_RESOLUTION</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Parquet"/>
+      <data name="Category" value="Schemas"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.6.0 IMPALA-2835 CDH-33330">
+      <indexterm audience="Cloudera">PARQUET_FALLBACK_SCHEMA_RESOLUTION query option</indexterm>
+      Allows Impala to look up columns within Parquet files by column name, rather than column order,
+      when necessary.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+    <p>
+      By default, Impala looks up columns within a Parquet file based on
+      the order of columns in the table.
+      The <codeph>name</codeph> setting for this option enables behavior for Impala queries
+      similar to the Hive setting <codeph>parquet.column.index.access=false</codeph>.
+      It also allows Impala to query Parquet files created by Hive with the
+      <codeph>parquet.column.index.access=false</codeph> setting in effect.
+    </p>
+
+    <p>
+      <b>Type:</b> integer or string.
+      Allowed values are 0 or <codeph>position</codeph> (default), 1 or <codeph>name</codeph>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/added_in_260"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_parquet.xml#parquet_schema_evolution"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_parquet_file_size.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_parquet_file_size.xml b/docs/topics/impala_parquet_file_size.xml
new file mode 100644
index 0000000..7019e93
--- /dev/null
+++ b/docs/topics/impala_parquet_file_size.xml
@@ -0,0 +1,86 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="parquet_block_size" id="parquet_file_size">
+
+  <title>PARQUET_FILE_SIZE Query Option</title>
+  <titlealts audience="PDF"><navtitle>PARQUET_FILE_SIZE</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Parquet"/>
+      <data name="Category" value="ETL"/>
+      <data name="Category" value="File Formats"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">PARQUET_FILE_SIZE query option</indexterm>
+      Specifies the maximum size of each Parquet data file produced by Impala <codeph>INSERT</codeph> statements.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+    <p>
+      Specify the size in bytes, or with a trailing <codeph>m</codeph> or <codeph>g</codeph> character to indicate
+      megabytes or gigabytes. For example:
+    </p>
+
+<codeblock>-- 128 megabytes.
+set PARQUET_FILE_SIZE=134217728
+INSERT OVERWRITE parquet_table SELECT * FROM text_table;
+
+-- 512 megabytes.
+set PARQUET_FILE_SIZE=512m;
+INSERT OVERWRITE parquet_table SELECT * FROM text_table;
+
+-- 1 gigabyte.
+set PARQUET_FILE_SIZE=1g;
+INSERT OVERWRITE parquet_table SELECT * FROM text_table;
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      With tables that are small or finely partitioned, the default Parquet block size (formerly 1 GB, now 256 MB
+      in Impala 2.0 and later) could be much larger than needed for each data file. For <codeph>INSERT</codeph>
+      operations into such tables, you can increase parallelism by specifying a smaller
+      <codeph>PARQUET_FILE_SIZE</codeph> value, resulting in more HDFS blocks that can be processed by different
+      nodes.
+<!-- Reducing the file size also reduces the memory required to buffer each block before writing it to disk. -->
+    </p>
+
+    <p>
+      <b>Type:</b> numeric, with optional unit specifier
+    </p>
+
+    <note type="important">
+    <p>
+      Currently, the maximum value for this setting is 1 gigabyte (<codeph>1g</codeph>).
+      Setting a value higher than 1 gigabyte could result in errors during
+      an <codeph>INSERT</codeph> operation.
+    </p>
+    </note>
+
+    <p>
+      <b>Default:</b> 0 (produces files with a target size of 256 MB; files might be larger for very wide tables)
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/isilon_blurb"/>
+    <p conref="../shared/impala_common.xml#common/isilon_block_size_caveat"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      For information about the Parquet file format, and how the number and size of data files affects query
+      performance, see <xref href="impala_parquet.xml#parquet"/>.
+    </p>
+
+<!-- Examples actually folded into Syntax earlier.   <p conref="../shared/impala_common.xml#common/example_blurb"/> -->
+
+  </conbody>
+</concept>


[28/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_fixed_issues.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_fixed_issues.xml b/docs/topics/impala_fixed_issues.xml
new file mode 100644
index 0000000..f55d95c
--- /dev/null
+++ b/docs/topics/impala_fixed_issues.xml
@@ -0,0 +1,7092 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="ver" id="fixed_issues">
+
+  <title><ph audience="standalone">Fixed Issues in Apache Impala (incubating)</ph><ph audience="integrated">Fixed Issues in Apache Impala (incubating)</ph></title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Release Notes"/>
+      <data name="Category" value="Fixed Issues"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Upgrading"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      The following sections describe the major issues fixed in each Impala release.
+    </p>
+
+    <p>
+      For known issues that are currently unresolved, see <xref href="impala_known_issues.xml#known_issues"/>.
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+<!-- All 2.7.x / 5.9.x subsections go under here -->
+
+  <concept rev="2.7.0" id="fixed_issues_270">
+
+    <title>Issues Fixed in Impala for CDH 5.9.0</title>
+
+    <conbody>
+
+      <p>
+<!--
+        The following list contains the most critical fixed issues
+        (<codeph>priority='Blocker'</codeph>) from the JIRA system.
+-->
+        For the full list of Impala fixed issues in Impala 2.7.0, see
+        <xref href="https://issues.cloudera.org/issues/?jql=%20type%20%3D%20bug%20and%20project%20%3D%20IMPALA%20AND%20resolution%20%3D%20fixed%20AND%20affectedVersion%20!%3D%20%22Impala%202.7.0%22%20AND%20fixVersion%20%3D%20%22Impala%202.7.0%22%20and%20not%20labels%20%3D%20broken-build%20order%20by%20priority%20desc" scope="external" format="html">this report in the Impala JIRA tracker</xref>.
+      </p>
+
+      <p> For the full list of fixed issues for all CDH components in CDH 5.9.0, see <xref
+          audience="standalone"
+          href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_59.html"
+          format="html" scope="external">the CDH 5.9.x release notes</xref><xref
+          audience="integrated" href="cdh_rn_fixed_in_59.xml"
+         />. </p>
+      <ul audience="integrated">
+        <li conref="cdh_rn_fixed_in_59.xml#ui_fixed_in_59/impala_start"
+          conrefend="cdh_rn_fixed_in_59.xml#ui_fixed_in_59/impala_end" />
+      </ul>
+
+    </conbody>
+  </concept>
+
+<!-- All 2.6.x / 5.8.x subsections go under here -->
+
+  <concept rev="2.6.3" id="fixed_issues_263">
+    <title>Issues Fixed in Impala for CDH 5.8.3</title>
+    <conbody>
+      <p> For the full list of fixed issues for all CDH components, see <xref
+          audience="standalone"
+          href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_58.html"
+          format="html" scope="external">the CDH 5.8.x release notes</xref><xref
+          audience="integrated" href="cdh_rn_fixed_in_58.xml#ui_fixed_in_583"
+         />. </p>
+      <ul audience="integrated">
+        <li conref="cdh_rn_fixed_in_58.xml#ui_fixed_in_583/impala_start"
+          conrefend="cdh_rn_fixed_in_58.xml#ui_fixed_in_583/impala_end" />
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept rev="2.6.2" id="fixed_issues_262">
+    <title>Issues Fixed in Impala for CDH 5.8.2</title>
+    <conbody>
+      <p> For the full list of fixed issues for all CDH components, see <xref
+          audience="standalone"
+          href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_58.html"
+          format="html" scope="external">the CDH 5.8.x release notes</xref><xref
+          audience="integrated" href="cdh_rn_fixed_in_58.xml#upstream582"
+         />. </p>
+      <ul audience="integrated">
+        <li conref="cdh_rn_fixed_in_58.xml#upstream582/impala_start"
+          conrefend="cdh_rn_fixed_in_58.xml#upstream582/impala_end" />
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept rev="2.6.0" id="fixed_issues_260">
+
+    <title>Issues Fixed in Impala for CDH 5.8.0</title>
+
+    <conbody>
+
+      <p>
+        The following list contains the most critical fixed issues
+        (<codeph>priority='Blocker'</codeph>) from the JIRA system.
+        For the full list of fixed issues in CDH 5.8.0 / Impala 2.6.0, see
+        <xref href="https://issues.cloudera.org/issues/?jql=%20type%20%3D%20bug%20and%20project%20%3D%20IMPALA%20AND%20resolution%20%3D%20fixed%20AND%20affectedVersion%20!%3D%20%22Impala%202.6.0%22%20AND%20fixVersion%20%3D%20%22Impala%202.6.0%22%20and%20not%20labels%20%3D%20broken-build%20order%20by%20priority%20desc" scope="external" format="html">this report in the Impala JIRA tracker</xref>.
+      </p>
+
+    </conbody>
+
+    <concept id="IMPALA-3385">
+      <title>RuntimeState::error_log_ crashes</title>
+      <conbody>
+      <p>
+        A crash could occur, with stack trace pointing to <codeph>impala::RuntimeState::ErrorLog</codeph>.
+      </p>
+      <p><b>Bug:</b> <xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3385">IMPALA-3385</xref></p>
+      <p><b>Severity:</b> High</p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-3378">
+      <title>HiveUdfCall::Open() produces unsynchronized access to JniUtil::global_refs_ vector</title>
+      <conbody>
+      <p>
+        A crash could occur because of contention between multiple calls to Java UDFs.
+      </p>
+      <p><b>Bug:</b> <xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3378">IMPALA-3378</xref></p>
+      <p><b>Severity:</b> High</p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-3379">
+      <title>HBaseTableWriter::CreatePutList() produces unsynchronized access to JniUtil::global_refs_ vector</title>
+      <conbody>
+      <p>
+        A crash could occur because of contention between multiple concurrent statements writing to HBase.
+      </p>
+      <p><b>Bug:</b> <xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3379">IMPALA-3379</xref></p>
+      <p><b>Severity:</b> High</p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-3317">
+      <title>Stress test failure: sorter.cc:745] Check failed: i == 0 (1 vs. 0) </title>
+      <conbody>
+      <p>
+        A crash or wrong results could occur if the spill-to-disk mechanism encountered a zero-length string at
+        the very end of a data block.
+      </p>
+      <p><b>Bug:</b> <xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3317">IMPALA-3317</xref></p>
+      <p><b>Severity:</b> High</p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-3311">
+      <title>String data coming out of agg can be corrupted by blocking operators</title>
+      <conbody>
+      <p>
+        If a query plan contains an aggregation node producing string values anywhere within a subplan
+        (that is,if in the SQL statement, the aggregate function appears within an inline view over a collection column),
+        the results of the aggregation may be incorrect.
+      </p>
+      <p><b>Bug:</b> <xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3311">IMPALA-3311</xref></p>
+      <p><b>Severity:</b> High</p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-3269">
+      <title>CTAS with subquery throws AuthzException</title>
+      <conbody>
+      <p>
+        A <codeph>CREATE TABLE AS SELECT</codeph> operation could fail with an authorization error,
+        due to a slight difference in the privilege checking for the CTAS operation.
+      </p>
+      <p><b>Bug:</b> <xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3269">IMPALA-3269</xref></p>
+      <p><b>Severity:</b> High</p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-3237">
+      <title>Crash on inserting into table with binary and parquet</title>
+      <conbody>
+      <p>
+        Impala incorrectly allowed <codeph>BINARY</codeph> to be specified as a column type,
+        resulting in a crash during a write to a Parquet table with a column of that type.
+      </p>
+      <p><b>Bug:</b> <xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3237">IMPALA-3237</xref></p>
+      <p><b>Severity:</b> High</p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-3105">
+      <title>RowBatch::MaxTupleBufferSize() calculation incorrect, may lead to memory corruption</title>
+      <conbody>
+      <p>
+        A crash could occur while querying tables with very large rows, for example wide tables with many
+        columns or very large string values. This problem was identified in Impala 2.3, but had low
+        reproducibility in subsequent releases. The fix ensures the memory allocation size is correct.
+      </p>
+      <p><b>Bug:</b> <xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3105">IMPALA-3105</xref></p>
+      <p><b>Severity:</b> High</p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-3494">
+      <title>Thrift buffer overflows when serialize more than 3355443200 bytes in impala</title>
+      <conbody>
+      <p>
+        A very large memory allocation within the <cmdname>catalogd</cmdname> daemon could exceed an internal Thrift limit,
+        causing a crash.
+      </p>
+      <p><b>Bug:</b> <xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3494">IMPALA-3494</xref></p>
+      <p><b>Severity:</b> High</p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-3314">
+      <title>Altering table partition's storage format is not working and crashing the daemon</title>
+      <conbody>
+      <p>
+        If a partitioned table used a file format other than Avro, and the file format of an individual partition
+        was changed to Avro, subsequent queries could encounter a crash.
+      </p>
+      <p><b>Bug:</b> <xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3314">IMPALA-3314</xref></p>
+      <p><b>Severity:</b> High</p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-3798">
+      <title>Race condition may cause scanners to spin with runtime filters on Avro or Sequence files</title>
+      <conbody>
+      <p>
+        A timing problem during runtime filter processing could cause queries against Avro or SequenceFile tables
+        to hang.
+      </p>
+      <p><b>Bug:</b> <xref scope="external" format="html" href="https://issues.cloudera.org/browse/IMPALA-3798">IMPALA-3798</xref></p>
+      <p><b>Severity:</b> High</p>
+      </conbody>
+    </concept>
+
+  </concept>
+<!-- All 2.5.x / 5.7.x subsections go under here -->
+
+  <concept rev="2.5.4" id="fixed_issues_254">
+    <title>Issues Fixed in Impala for CDH 5.7.4</title>
+    <conbody>
+      <p> For the full list of fixed issues for all CDH components, see <xref
+          audience="standalone"
+          href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_57.html"
+          format="html" scope="external">the CDH 5.7.x release notes</xref><xref
+          audience="integrated" href="cdh_rn_fixed_in_57.xml#ui_fixed_in_574"
+         />. </p>
+      <ul audience="integrated">
+        <li conref="cdh_rn_fixed_in_57.xml#ui_fixed_in_574/impala_start"
+          conrefend="cdh_rn_fixed_in_57.xml#ui_fixed_in_574/impala_end" />
+      </ul>
+    </conbody>
+  </concept>
+
+<!--
+  <concept rev="2.5.3" id="fixed_issues_253" audience="pre-release">
+
+    <title>Issues Fixed in Impala for CDH 5.7.3</title>
+
+    <conbody>
+
+      <p>
+        For the full list of fixed issues for all CDH components, see
+        <xref audience="standalone"
+          href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_57.html#ui_fixed_in_573"
+          format="html" scope="external">the CDH 5.7.x release notes</xref><xref
+            audience="integrated" href="cdh_rn_fixed_in_57.xml#ui_fixed_in_573"/>.
+      </p>
+
+      <ul audience="integrated">
+        <li conref="cdh_rn_fixed_in_57.xml#ui_fixed_in_573/impala_start" conrefend="cdh_rn_fixed_in_57.xml#ui_fixed_in_573/impala_end"/>
+      </ul>
+
+    </conbody>
+
+  </concept>
+-->
+
+  <concept rev="2.5.2" id="fixed_issues_252">
+    <title>Issues Fixed in Impala for CDH 5.7.2</title>
+    <conbody>
+      <p> For the full list of fixed issues for all CDH components, see <xref
+          audience="standalone"
+          href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_57.html#ui_fixed_in_572"
+          format="html" scope="external">the CDH 5.7.x release notes</xref><xref
+          audience="integrated" href="cdh_rn_fixed_in_57.xml#ui_fixed_in_572"
+         />. </p>
+      <ul audience="integrated">
+        <li conref="cdh_rn_fixed_in_57.xml#ui_fixed_in_572/impala_start"
+          conrefend="cdh_rn_fixed_in_57.xml#ui_fixed_in_572/impala_end" />
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept rev="2.5.1" id="fixed_issues_251">
+
+    <title>Issues Fixed in Impala for CDH 5.7.1</title>
+
+    <conbody>
+
+      <p>
+        For the full list of fixed issues for all CDH components, see
+        <xref audience="standalone"
+          href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_57.html#ui_fixed_in_571"
+          format="html" scope="external">the CDH 5.7.x release notes</xref><xref
+            audience="integrated" href="cdh_rn_fixed_in_57.xml#ui_fixed_in_571"/>.
+      </p>
+
+      <ul audience="integrated">
+        <li conref="cdh_rn_fixed_in_57.xml#ui_fixed_in_571/impala_start" conrefend="cdh_rn_fixed_in_57.xml#ui_fixed_in_571/impala_end"/>
+      </ul>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.5.0" id="fixed_issues_250">
+
+    <title>Issues Fixed in Impala for CDH 5.7.0</title>
+
+    <conbody>
+
+      <p>
+        The following list contains the most critical issues (<codeph>priority='Blocker'</codeph>) from the JIRA system.
+        For the full list of fixed issues in CDH 5.7.0 / Impala 2.5.0, see
+        <xref href="https://issues.cloudera.org/issues/?jql=%20type%20%3D%20bug%20and%20project%20%3D%20IMPALA%20AND%20resolution%20%3D%20fixed%20AND%20affectedVersion%20!%3D%20%22Impala%202.5.0%22%20AND%20fixVersion%20%3D%20%22Impala%202.5.0%22%20and%20not%20labels%20%3D%20broken-build%20order%20by%20priority%20desc" scope="external" format="html">this report in the Impala JIRA tracker</xref>.
+      </p>
+
+    </conbody>
+
+    <concept id="IMPALA-2683">
+      <title>Stress test hit assert in LLVM: external function could not be resolved</title>
+      <conbody>
+<p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2683" scope="external" format="html">IMPALA-2683</xref></p>
+<p>The stress test was running a build with the TPC-H, TPC-DS, and TPC-H nested queries with scale factor 3.</p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2365">
+      <title>Impalad is crashing if udf jar is not available in hdfs location for first time</title>
+      <conbody>
+        <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2365" scope="external" format="html">IMPALA-2365</xref></p>
+        <p>
+          If a UDF JAR was not available in the HDFS location specified in the <codeph>CREATE FUNCTION</codeph> statement,
+          the <cmdname>impalad</cmdname> daemon could crash.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2535-570">
+      <title>PAGG hits mem_limit when switching to I/O buffers</title>
+      <conbody>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2535" scope="external" format="html">IMPALA-2535</xref></p>
+      <p>
+        A join query could fail with an out-of-memory error despite the apparent presence of sufficient memory.
+        The cause was the internal ordering of operations that could cause a later phase of the query to
+        allocate memory required by an earlier phase of the query. The workaround was to either increase
+        or decrease the <codeph>MEM_LIMIT</codeph> query option, because the issue would only occur for a specific
+        combination of memory limit and data volume.
+      </p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2643-570">
+      <title>Prevent migrating incorrectly inferred identity predicates into inline views</title>
+      <conbody>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2643" scope="external" format="html">IMPALA-2643</xref></p>
+      <p>
+        Referring to the same column twice in a view definition could cause the view to omit
+        rows where that column contained a <codeph>NULL</codeph> value. This could cause
+        incorrect results due to an inaccurate <codeph>COUNT(*)</codeph> value or rows missing
+        from the result set.
+      </p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-1459-570">
+      <title>Fix migration/assignment of On-clause predicates inside inline views</title>
+      <conbody>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1459" scope="external" format="html">IMPALA-1459</xref></p>
+      <p>
+        Some combinations of <codeph>ON</codeph> clauses in join queries could result in comparisons
+        being applied at the wrong stage of query processing, leading to incorrect results.
+        Wrong predicate assignment could happen under the following conditions:
+      </p>
+      <ul>
+        <li>
+          The query includes an inline view that contains an outer join.
+        </li>
+        <li>
+          That inline view is joined with another table in the enclosing query block.
+        </li>
+        <li>
+          That join has an <codeph>ON</codeph> clause containing a predicate that
+          only references columns originating from the outer-joined tables inside the inline view.
+        </li>
+      </ul>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2093">
+      <title>Wrong plan of NOT IN aggregate subquery when a constant is used in subquery predicate</title>
+      <conbody>
+        <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2093" scope="external" format="html">IMPALA-2093</xref></p>
+        <p>
+          <codeph>IN</codeph> subqueries might return wrong results if the left-hand side of the <codeph>IN</codeph> is a constant.
+          For example:
+        </p>
+<codeblock>
+select * from alltypestiny t1
+  where 10 not in (select sum(int_col) from alltypestiny);
+</codeblock>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2940">
+      <title>Parquet DictDecoders accumulate throughout query</title>
+      <conbody>
+        <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2940" scope="external" format="html">IMPALA-2940</xref></p>
+        <p>
+          Parquet dictionary decoders can accumulate throughout query execution, leading to excessive memory usage. One decoder is created per-column per-split.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-3056">
+      <title>Planner doesn't set the has_local_target field correctly</title>
+      <conbody>
+<p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-3056" scope="external" format="html">IMPALA-3056</xref></p>
+
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2742">
+      <title>MemPool allocation growth behavior</title>
+      <conbody>
+        <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2742" scope="external" format="html">IMPALA-2742</xref></p>
+        <p>
+          Currently, the MemPool would always double the size of the last allocation.
+          This can lead to bad behavior if the MemPool transferred the ownership of all its data
+          except the last chunk. In the next allocation, the next allocated chunk would double
+          the size of this large chunk, which can be undesirable.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-3035">
+      <title>Drop partition operations don't follow the catalog's locking protocol</title>
+      <conbody>
+        <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-3035" scope="external" format="html">IMPALA-3035</xref></p>
+        <p>
+          The <codeph>CatalogOpExecutor.alterTableDropPartition()</codeph> function violates
+          the locking protocol used in the catalog that requires <codeph>catalogLock_</codeph>
+          to be acquired before any table-level lock. That may cause deadlocks when <codeph>ALTER TABLE DROP PARTITION</codeph>
+          is executed concurrently with other DDL operations.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2215">
+      <title>HAVING clause without aggregation not applied properly</title>
+      <conbody>
+        <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2215" scope="external" format="html">IMPALA-2215</xref></p>
+        <p>
+          A query with a <codeph>HAVING</codeph> clause but no <codeph>GROUP BY</codeph> clause was not being rejected,
+          despite being invalid syntax. For example:
+        </p>
+
+<codeblock>
+select case when 1=1 then 'didit' end as c1 from (select 1 as one) a having 1!=1;
+</codeblock>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2914">
+      <title>Hit DCHECK Check failed: HasDateOrTime()</title>
+      <conbody>
+        <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2914" scope="external" format="html">IMPALA-2914</xref></p>
+        <p>
+          <codeph>TimestampValue::ToTimestampVal()</codeph> requires a valid <codeph>TimestampValue</codeph> as input.
+          This requirement was not enforced in some places, leading to serious errors.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2986">
+      <title>Aggregation spill loop gives up too early leading to mem limit exceeded errors</title>
+      <conbody>
+        <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2986" scope="external" format="html">IMPALA-2986</xref></p>
+        <p>
+          An aggregation query could fail with an out-of-memory error, despite sufficient memory being reported as available.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2592">
+      <title>DataStreamSender::Channel::CloseInternal() does not close the channel on an error.</title>
+      <conbody>
+        <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2592" scope="external" format="html">IMPALA-2592</xref></p>
+        <p>
+          Some queries do not close an internal communication channel on an error.
+          This will cause the node on the other side of the channel to wait indefinitely, causing the query to hang.
+          For example, this issue could happen on a Kerberos-enabled system if the credential cache was outdated.
+          Although the affected query hangs, the <cmdname>impalad</cmdname> daemons continue processing other queries.
+        </p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2184">
+      <title>Codegen does not catch exceptions in FROM_UNIXTIME()</title>
+      <conbody>
+        <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2184" scope="external" format="html">IMPALA-2184</xref></p>
+        <p>
+          Querying for the min or max value of a timestamp cast from a bigint via <codeph>from_unixtime()</codeph>
+          fails silently and crashes instances of <cmdname>impalad</cmdname> when the input includes a value outside of the valid range.
+        </p>
+
+        <p><b>Workaround:</b> Disable native code generation with:</p>
+<codeblock>
+SET disable_codegen=true;
+</codeblock>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2788">
+      <title>Impala returns wrong result for function 'conv(bigint, from_base, to_base)'</title>
+      <conbody>
+        <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2788" scope="external" format="html">IMPALA-2788</xref></p>
+        <p>
+          Impala returns wrong result for function <codeph>conv()</codeph>.
+          Function <codeph>conv(bigint, from_base, to_base)</codeph> returns an correct result,
+          while <codeph>conv(string, from_base, to_base)</codeph> returns the correct value.
+          For example:
+        </p>
+
+<codeblock>
+<![CDATA[
+select 2061013007, conv(2061013007, 16, 10), conv('2061013007', 16, 10);
++------------+--------------------------+----------------------------+
+| 2061013007 | conv(2061013007, 16, 10) | conv('2061013007', 16, 10) |
++------------+--------------------------+----------------------------+
+| 2061013007 | 1627467783               | 139066421255               |
++------------+--------------------------+----------------------------+
+Fetched 1 row(s) in 0.65s
+
+select 2061013007, conv(cast(2061013007 as bigint), 16, 10), conv('2061013007', 16, 10);
++------------+------------------------------------------+----------------------------+
+| 2061013007 | conv(cast(2061013007 as bigint), 16, 10) | conv('2061013007', 16, 10) |
++------------+------------------------------------------+----------------------------+
+| 2061013007 | 1627467783                               | 139066421255               |
++------------+------------------------------------------+----------------------------+
+
+select 2061013007, conv(cast(2061013007 as string), 16, 10), conv('2061013007', 16, 10);
++------------+------------------------------------------+----------------------------+
+| 2061013007 | conv(cast(2061013007 as string), 16, 10) | conv('2061013007', 16, 10) |
++------------+------------------------------------------+----------------------------+
+| 2061013007 | 139066421255                             | 139066421255               |
++------------+------------------------------------------+----------------------------+
+
+select 2061013007, conv(cast(cast(2061013007 as decimal(20,0)) as bigint), 16, 10), conv('2061013007', 16, 10);
++------------+-----------------------------------------------------------------+----------------------------+
+| 2061013007 | conv(cast(cast(2061013007 as decimal(20,0)) as bigint), 16, 10) | conv('2061013007', 16, 10) |
++------------+-----------------------------------------------------------------+----------------------------+
+| 2061013007 | 1627467783                                                      | 139066421255               |
++------------+-----------------------------------------------------------------+----------------------------+
+]]>
+</codeblock>
+
+        <p><b>Workaround:</b>
+          Cast the value to string and use <codeph>conv(string, from_base, to_base)</codeph> for conversion.
+        </p>
+      </conbody>
+    </concept>
+
+<!-- I think this issue is marked incorrectly in JIRA. It arose during the 5.7 development cycle, it's not a
+     customer-facing issue.
+    <concept id="IMPALA-3039">
+      <title>Restrict the number of runtime filters generated</title>
+      <conbody>
+        <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-3039" scope="external" format="html">IMPALA-3039</xref></p>
+        <p>
+          Large queries with many runtime filters might fail for various reasons, such as memory for Thrift data structures
+          not being accounted for.
+          Having to disable all runtime filters to make a query run is not desirable.
+          The new <codeph>MAX_NUM_RUNTIME_FILTERS</codeph> query option specifies an upper bound
+          on the number of runtime filters generated for a query.
+        </p>
+      </conbody>
+    </concept>
+-->
+
+  </concept>
+
+<!-- All 2.4.x / 5.6.x subsections go under here -->
+
+  <concept rev="2.4.1" id="fixed_issues_241">
+
+    <title>Issues Fixed in Impala for CDH 5.6.1</title>
+
+    <conbody>
+
+      <p>
+        For the full list of fixed issues for all CDH components, see
+        <xref audience="standalone"
+          href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_56.html#ui_fixed_in_561"
+          format="html" scope="external">the CDH 5.6.x release notes</xref><xref
+            audience="integrated" href="cdh_rn_fixed_in_56.xml#ui_fixed_in_561"/>.
+      </p>
+
+      <!-- Maybe time to start dropping the 'CDH 5 only' notes... wasn't included for 5.6.0, so why start for 5.6.1...
+           <note conref="../shared/impala_common.xml#common/only_cdh5_24x"/> -->
+
+      <ul audience="integrated">
+        <li conref="cdh_rn_fixed_in_56.xml#ui_fixed_in_561/impala_start" conrefend="cdh_rn_fixed_in_56.xml#ui_fixed_in_561/impala_end"/>
+      </ul>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.4.0" id="fixed_issues_240">
+
+    <title>Issues Fixed in Impala for CDH 5.6.0</title>
+
+    <conbody>
+
+      <p> The set of fixes for Impala in CDH 5.6.0 is the same as in CDH 5.5.2.
+        See <xref href="impala_fixed_issues.xml#fixed_issues_232"
+          audience="integrated"/> for details. </p>
+
+<!--
+      <p>
+        This section lists the most serious or frequently encountered customer issues fixed in CDH 5.6.0 / Impala 2.4.0.
+        For the full list of fixed Impala issues, see
+        <xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_560.html" format="html" scope="external">the CDH 5.6.x release notes</xref><xref audience="integrated" href="cdh_rn_fixed_in_560.xml"/>.
+      </p>
+-->
+
+    </conbody>
+
+  </concept>
+
+<!-- All 2.3.x / 5.5.x subsections go under here -->
+
+  <concept rev="2.3.4" id="fixed_issues_234">
+
+    <title>Issues Fixed in Impala for CDH 5.5.4</title>
+
+    <conbody>
+
+      <p>
+        For the full list of fixed issues for all CDH components, see
+        <xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_55.html#fixed_issues_554" format="html" scope="external">the CDH 5.5.x release notes</xref><xref audience="integrated" href="cdh_rn_fixed_in_55.xml#fixed_issues_554"/>.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_23x"/>
+
+      <ul audience="integrated">
+        <li conref="cdh_rn_fixed_in_55.xml#ui_fixed_in_554/impala_start" conrefend="cdh_rn_fixed_in_55.xml#ui_fixed_in_554/impala_end"/>
+      </ul>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.3.2" id="fixed_issues_232">
+
+    <title>Issues Fixed in Impala for CDH 5.5.2</title>
+
+    <conbody>
+
+      <p>
+        This section lists the most serious or frequently encountered customer
+        issues fixed in CDH 5.5.2 / Impala 2.3.2. For the full list of fixed Impala issues,
+        see <xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_552.html" format="html" scope="external">the CDH 5.5.x release notes</xref><xref href="cdh_rn_fixed_in_55.xml#fixed_issues_552" audience="integrated"/>.
+      </p>
+
+    </conbody>
+
+<!--
+Fixed issues in 5.5.2 from Juan Yu spreadsheet (https://docs.google.com/spreadsheets/d/1Ia06xOLKQInKfsJzQfQ6Sf1fO3nYLOpZ-0OWfFNgNSc/edit#gid=2094272728):
+
+IMPALA-2747: Thrift-client cleans openSSL state before using it in the case of the catalog
+IMPALA-2829: SEGV in AnalyticEvalNode touching NULL input_stream_
+IMPALA-2722: Free local allocations per row batch in non-partitioned AGG and HJ
+IMPALA-2612: Free local allocations once for every row batch when building hash tables
+IMPALA-2643: Prevent migrating incorrectly inferred identity predicates into inline views
+IMPALA-2695: Fix GRANTs on URIs with uppercase letters
+IMPALA-2664: Avoid sending large partition stats objects over thrift
+IMPALA-2648: Ditto
+IMPALA-2226: Throw AnalysisError if table properties are too large (for the Hive metastore)
+IMPALA-2273: Make MAX_PAGE_HEADER_SIZE configurable
+IMPALA-2473: reduce scanner memory usage
+IMPALA-2113: Handle error when distinct and aggregates are used with a having clause
+IMPALA-2225: Handle error when star based select item and aggregate are incorrectly used
+IMPALA-2731: Refactor MemPool usage in HBase scan node
+IMPALA-1459: Fix migration/assignment of On-clause predicates inside inline views
+IMPALA-2558: DCHECK in parquet scanner after block read error
+IMPALA-2535: PAGG hits mem_limit when switching to I/O buffers
+IMPALA-2559: Fix check failed: sorter_runs_.back()->is_pinned_
+IMPALA-2614: Don't ignore Status returned by DataStreamRecvr::CreateMerger()
+IMPALA-2591: DataStreamSender::Send() does not return an error status if SendBatch() failed
+IMPALA-2598: Re-enable SSL and Kerberos on server-server
+
+These are changes to test cases and so ignored for release note purposes:
+
+IMPALA-2103: Fix flaky test_impersonation test
+IMPALA-2776: Remove escapechartesttable and associated tests.
+IMPALA-2812: Remove additional test referencing escapecharstesttable.
+IMPALA-2630: skip TestParquet.test_continue_on_error when using old aggs/joins
+IMPALA-2624: Increase fs.trash.interval to 24 hours for test suite
+-->
+
+    <concept id="IMPALA-2829">
+      <title>SEGV in AnalyticEvalNode touching NULL input_stream_</title>
+      <conbody>
+      <p>
+        A query involving an analytic function could encounter a serious error.
+        This issue was encountered infrequently, depending upon specific combinations
+        of queries and data.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2829" scope="external" format="html">IMPALA-2829</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2722">
+      <title>Free local allocations per row batch in non-partitioned AGG and HJ</title>
+      <conbody>
+      <p>
+        An outer join query could fail unexpectedly with an out-of-memory error
+        when the <q>spill to disk</q> mechanism was turned off.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2722" scope="external" format="html">IMPALA-2722</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2612">
+      <!-- In this case, the title from the spreadsheet is clearer than the original JIRA title. -->
+      <title>Free local allocations once for every row batch when building hash tables</title>
+      <conbody>
+      <p>
+        A join query could encounter a serious error due to an internal failure to allocate memory, which
+        resulted in dereferencing a <codeph>NULL</codeph> pointer.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2612" scope="external" format="html">IMPALA-2612</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2643">
+      <title>Prevent migrating incorrectly inferred identity predicates into inline views</title>
+      <conbody>
+      <p>
+        Referring to the same column twice in a view definition could cause the view to omit
+        rows where that column contained a <codeph>NULL</codeph> value. This could cause
+        incorrect results due to an inaccurate <codeph>COUNT(*)</codeph> value or rows missing
+        from the result set.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2643" scope="external" format="html">IMPALA-2643</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2695">
+      <title>Fix GRANTs on URIs with uppercase letters</title>
+      <conbody>
+      <p>
+        A <codeph>GRANT</codeph> statement for a URI could be ineffective if the URI
+        contained uppercase letters, for example in an uppercase directory name.
+        Subsequent statements, such as <codeph>CREATE EXTERNAL TABLE</codeph>
+        with a <codeph>LOCATION</codeph> clause, could fail with an authorization exception.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2695" scope="external" format="html">IMPALA-2695</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2664-552">
+      <title id="IMPALA-2648-552">Avoid sending large partition stats objects over thrift</title>
+      <conbody>
+      <p>
+        The <cmdname>catalogd</cmdname> daemon could encounter a serious error
+        when loading the incremental statistics metadata for tables with large
+        numbers of partitions and columns. The problem occurred when the
+        internal representation of metadata for the table exceeded 2
+        GB, for example in a table with 20K partitions and 77 columns. The fix causes a
+        <codeph>COMPUTE INCREMENTAL STATS</codeph> operation to fail if it
+        would produce metadata that exceeded the maximum size.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2664" scope="external" format="html">IMPALA-2664</xref>,
+        <xref href="https://issues.cloudera.org/browse/IMPALA-2648" scope="external" format="html">IMPALA-2648</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2226">
+      <title>Throw AnalysisError if table properties are too large (for the Hive metastore)</title>
+      <conbody>
+      <p>
+        <codeph>CREATE TABLE</codeph> or <codeph>ALTER TABLE</codeph> statements could fail with
+        metastore database errors due to length limits on the <codeph>SERDEPROPERTIES</codeph> and <codeph>TBLPROPERTIES</codeph> clauses.
+        (The limit on key size is 256, while the limit on value size is 4000.) The fix makes Impala handle these error conditions
+        more cleanly, by detecting too-long values rather than passing them to the metastore database.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2226" scope="external" format="html">IMPALA-2226</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2273-552">
+      <title>Make MAX_PAGE_HEADER_SIZE configurable</title>
+      <conbody>
+      <p>
+        Impala could fail to access Parquet data files with page headers larger than 8 MB, which could
+        occur, for example, if the minimum or maximum values for a column were long strings. The
+        fix adds a configuration setting <codeph>--max_page_header_size</codeph>, which you can use to
+        increase the Impala size limit to a value higher than 8 MB.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2273" scope="external" format="html">IMPALA-2273</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2473">
+      <title>reduce scanner memory usage</title>
+      <conbody>
+      <p>
+        Queries on Parquet tables could consume excessive memory (potentially multiple gigabytes) due to producing
+        large intermediate data values while evaluating groups of rows. The workaround was to reduce the size of
+        the <codeph>NUM_SCANNER_THREADS</codeph> query option, the <codeph>BATCH_SIZE</codeph> query option,
+        or both.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2473" scope="external" format="html">IMPALA-2473</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2113">
+      <title>Handle error when distinct and aggregates are used with a having clause</title>
+      <conbody>
+      <p>
+        A query that included a <codeph>DISTINCT</codeph> operator and a <codeph>HAVING</codeph> clause, but no
+        aggregate functions or <codeph>GROUP BY</codeph>, would fail with an uninformative error message.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2113" scope="external" format="html">IMPALA-2113</xref></p>
+      <!-- Don't think this is really a 'major' issue. Left a comment in the IMPALA JIRA. -->
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2225">
+      <title>Handle error when star based select item and aggregate are incorrectly used</title>
+      <conbody>
+      <p>
+        A query that included <codeph>*</codeph> in the <codeph>SELECT</codeph> list, in addition to an
+        aggregate function call, would fail with an uninformative message if the query had no
+        <codeph>GROUP BY</codeph> clause.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2225" scope="external" format="html">IMPALA-2225</xref></p>
+      <!-- Don't think this is really a 'major' issue. Left a comment in the IMPALA JIRA. -->
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2731-552">
+      <title>Refactor MemPool usage in HBase scan node</title>
+      <conbody>
+      <p>
+        Queries involving HBase tables used substantially more memory than in earlier Impala versions.
+        The problem occurred starting in Impala 2.2.8, as a result of the changes for IMPALA-2284.
+        The fix for this issue involves removing a separate memory work area for HBase queries
+        and reusing other memory that was already allocated.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2731" scope="external" format="html">IMPALA-2731</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-1459-552">
+      <title>Fix migration/assignment of On-clause predicates inside inline views</title>
+      <conbody>
+      <p>
+        Some combinations of <codeph>ON</codeph> clauses in join queries could result in comparisons
+        being applied at the wrong stage of query processing, leading to incorrect results.
+        Wrong predicate assignment could happen under the following conditions:
+      </p>
+      <ul>
+        <li>
+          The query includes an inline view that contains an outer join.
+        </li>
+        <li>
+          That inline view is joined with another table in the enclosing query block.
+        </li>
+        <li>
+          That join has an <codeph>ON</codeph> clause containing a predicate that
+          only references columns originating from the outer-joined tables inside the inline view.
+        </li>
+      </ul>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1459" scope="external" format="html">IMPALA-1459</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2558">
+      <title>DCHECK in parquet scanner after block read error</title>
+      <conbody>
+      <p>
+        A debug build of Impala could encounter a serious error after encountering some kinds of I/O
+        errors for Parquet files. This issue only occurred in debug builds, not release builds.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2558" scope="external" format="html">IMPALA-2558</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2535">
+      <title>PAGG hits mem_limit when switching to I/O buffers</title>
+      <conbody>
+      <p>
+        A join query could fail with an out-of-memory error despite the apparent presence of sufficient memory.
+        The cause was the internal ordering of operations that could cause a later phase of the query to
+        allocate memory required by an earlier phase of the query. The workaround was to either increase
+        or decrease the <codeph>MEM_LIMIT</codeph> query option, because the issue would only occur for a specific
+        combination of memory limit and data volume.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2535" scope="external" format="html">IMPALA-2535</xref></p>
+      </conbody>
+    </concept>
+
+    <!-- This is a pretty incomprehensible JIRA title but there isn't any clearer title in the spreadsheet. -->
+    <!-- Merged with the following JIRA entry (which has a more understandable title) because it's part of the same root cause. -->
+    <concept id="IMPALA-2559">
+      <title>Fix check failed: sorter_runs_.back()->is_pinned_</title>
+      <conbody>
+      <p>
+        A query could fail with an internal error while calculating the memory limit.
+        This was an infrequent condition uncovered during stress testing.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2559" scope="external" format="html">IMPALA-2559</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2614">
+      <title>Don't ignore Status returned by DataStreamRecvr::CreateMerger()</title>
+      <conbody>
+      <p>
+        A query could fail with an internal error while calculating the memory limit.
+        This was an infrequent condition uncovered during stress testing.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2614" scope="external" format="html">IMPALA-2614</xref>,
+      <xref href="https://issues.cloudera.org/browse/IMPALA-2559" scope="external" format="html">IMPALA-2559</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2591">
+      <title>DataStreamSender::Send() does not return an error status if SendBatch() failed</title>
+      <conbody>
+      <!-- Symptoms listed in the JIRA are very vague. Left a note asking Sailesh for more detail. -->
+      <p>
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2591" scope="external" format="html">IMPALA-2591</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2598">
+      <title>Re-enable SSL and Kerberos on server-server</title>
+      <conbody>
+      <p>
+        These fixes lift the restriction on using SSL encryption and Kerberos authentication together
+        for internal communication between Impala components.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2598" scope="external" format="html">IMPALA-2598</xref>,
+      <xref href="https://issues.cloudera.org/browse/IMPALA-2747" scope="external" format="html">IMPALA-2747</xref></p>
+      </conbody>
+    </concept>
+
+<!--
+    <concept id="IMPALA-2747">
+      <title>Thrift-client cleans openSSL state before using it in the case of the catalog</title>
+      <conbody>
+      <p></p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2747" scope="external" format="html">IMPALA-2747</xref></p>
+      <p><b>Severity:</b> High</p>
+      </conbody>
+    </concept>
+-->
+
+  </concept>
+
+  <concept rev="2.3.1" id="fixed_issues_231">
+
+    <title>Issues Fixed in Impala for CDH 5.5.1</title>
+
+    <conbody>
+
+      <p conref="../shared/impala_common.xml#common/cdh551_noop"/>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.3.0" id="fixed_issues_230">
+
+    <title>Issues Fixed in Impala for CDH 5.5.0</title>
+
+    <conbody>
+      <p> This section lists the most serious or frequently encountered customer
+        issues fixed in CDH 5.5.0 / Impala 2.3.0. Any issues already fixed in
+        CDH 5.4 maintenance releases (up through CDH 5.4.8) are also included.
+        Those issues are listed under the respective CDH 5.4 sections and are
+        not repeated here. For the full list of fixed Impala issues, see <xref
+          audience="standalone"
+          href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_550.html"
+          format="html" scope="external">the CDH 5.5.x release notes</xref><xref
+          href="cdh_rn_fixed_in_55.xml#concept_hjb_1pf_x5" audience="integrated"
+        />. </p>
+
+<!-- Raw "Fixed Issue" info from Silvius' working doc: https://docs.google.com/document/d/1PCboHo2Em-fySAHr3YF1ZZxMEfK8_vFFpP1rkWzyRZM/edit#
+
+Bug
+IMPALA-2168
+SEGV in BufferedTupleStream::num_rows() in a query with very large, spilling ROJ
+crash, resource-management
+
+Bug
+IMPALA-2213
+Parquet read can fail if file metadata is stale
+
+Bug
+IMPALA-2192
+Wrong results on TPCH-Q11 during nightly data load
+broken-build, correctness
+
+Bug
+IMPALA-2330
+SwitchToIoBuffers may leave stream to wrong state
+resource-management
+
+Bug
+IMPALA-2440
+Full outer join using non-partitioned HJ can incorrectly produce extra row of nulls
+correctness
+
+Bug
+IMPALA-2435
+Sorter::AddBatch does not check the Status of unsorted_run_->Init()
+
+Bug
+IMPALA-2378
+Impalad exceeded its mem limit
+crash, resource-management
+
+Bug
+IMPALA-2369
+Crash: impala::Sorter::Run::Init
+crash, nested_types, query_generator
+
+Bug
+IMPALA-2366
+DiskIoMgr error handling with fread()
+
+Bug
+IMPALA-2357
+Crash: impala::BufferedBlockMgr::Block::BytesRemaining
+crash, nested_types, query_generator
+
+Bug
+IMPALA-2348
+The Catalog does not close the connection to the HMS after table invalidation
+catalog-server, hive
+
+Bug
+IMPALA-2319
+Crash with nested loop join + limit
+
+Bug
+IMPALA-2314
+Crash in PHJ::PrepareNextPartition() in call to LargestSpilledPartition()
+
+Bug
+IMPALA-2165
+Avoid estimating cardinality 0 in SCAN node
+performance, planner, resource-management
+
+Improvement
+IMPALA-2143
+Avoid impala-shell sending password in plain text with ssl enabled
+
+Bug
+IMPALA-2133
+HBase filter doesn't unescape string values correctly
+
+Bug
+IMPALA-2090
+Incorrect handling of leap years when adding 1 month interval to date
+correctness, query_generator
+
+Bug
+IMPALA-2086
+Incorrect handling of leap years when adding 1 year interval to date
+correctness, query_generator
+
+New Feature
+IMPALA-2084
+SPLIT_PART and REGEXP_LIKE functions for Tableau pushdown
+
+New Feature
+IMPALA-2081
+Add PERCENT_RANK, NTILE, CUME_DIST analytic window functions
+planner, ramp-up-planner
+
+Bug
+IMPALA-2016
+Cancelling query with group_concat causes crash
+crash, query-lifecycle
+
+Improvement
+IMPALA-1975
+Automatically reconnect to Impala in the shell if the connection is lost
+ramp-up-task, shell
+
+Improvement - perf
+IMPALA-1968
+Bad plan choices due to incorrect number of estimated hosts.
+performance
+
+Bug
+IMPALA-1947
+Avro cols may load incorrectly if schema inconsistent with StorageDescriptor
+catalog-server, correctness
+
+Bug
+IMPALA-1917
+Query return empty result if it contains NullLiteral in inlineview
+correctness, regression
+
+(Possible new feature entry?)
+Improvement - perf
+IMPALA-1881
+Maximize data locality when scanning Parquet files with multiple row groups.
+parquet, performance
+
+(Possible new feature entry?)
+Improvement
+IMPALA-1870
+Support users containing commas in authorized_proxy_user_config
+ramp-up-introductory, ramp-up-task
+
+Bug
+IMPALA-1859
+INSERT OVERWRITE with empty result set leaves existing records in partitioned table
+impala, insert, overwrite, partitions
+
+Bug
+IMPALA-1813
+CREATE TABLE LIKE ... STORED AS AVRO does not work.
+usability
+
+Improvement
+IMPALA-1136
+Impala is unable to read hive tables created with the "STORED AS AVRO" clause
+ramp-up-planner, ramp-up-task, usability
+
+(Possible new feature entry?)
+Improvement - perf
+IMPALA-1588
+Cache HDFS file handle to avoid repeated hdfs fopen call
+
+Bug
+IMPALA-1675
+Timestamp: Adding/subtracting very large time intervals to timestamps produces incorrect result
+
+Improvement
+IMPALA-1982
+parquet.hive.serde.ParquetHiveSerDe is deprecated
+Skye Wanderman-Milne
+Critical
+supportability
+
+Bug
+IMPALA-1906
+PARQUET_FILE_SIZE query option not always honored due to an internal miscalculation.
+Vlad Berindei
+Critical
+parquet, ramp-up-introductory, ramp-up-task
+
+-->
+
+    </conbody>
+
+    <concept id="serious_230">
+      <title>Fixes for Serious Errors</title>
+      <conbody>
+      <p>
+        A number of issues were resolved that could result in serious errors
+        when encountered. The most critical or commonly encountered are
+        listed here.
+      </p>
+      <p><b>Bugs:</b>
+
+<!-- Bugs marked as 'crash'.
+IMPALA-2168
+IMPALA-2378
+IMPALA-2369
+IMPALA-2357
+IMPALA-2319
+IMPALA-2314
+IMPALA-2016
+-->
+
+      <xref href="https://issues.cloudera.org/browse/IMPALA-2168" scope="external" format="html">IMPALA-2168</xref>,
+      <xref href="https://issues.cloudera.org/browse/IMPALA-2378" scope="external" format="html">IMPALA-2378</xref>,
+      <xref href="https://issues.cloudera.org/browse/IMPALA-2369" scope="external" format="html">IMPALA-2369</xref>,
+      <xref href="https://issues.cloudera.org/browse/IMPALA-2357" scope="external" format="html">IMPALA-2357</xref>,
+      <xref href="https://issues.cloudera.org/browse/IMPALA-2319" scope="external" format="html">IMPALA-2319</xref>,
+      <xref href="https://issues.cloudera.org/browse/IMPALA-2314" scope="external" format="html">IMPALA-2314</xref>,
+      <xref href="https://issues.cloudera.org/browse/IMPALA-2016" scope="external" format="html">IMPALA-2016</xref>
+      </p>
+      </conbody>
+    </concept>
+
+    <concept id="correctness_230">
+      <title>Fixes for Correctness Errors</title>
+      <conbody>
+      <p>
+        A number of issues were resolved that could result in wrong results
+        when encountered. The most critical or commonly encountered are
+        listed here.
+      </p>
+      <p><b>Bugs:</b>
+
+<!-- Bugs marked as 'correctness'.
+IMPALA-2192
+IMPALA-2440
+IMPALA-2090
+IMPALA-2086
+IMPALA-1947
+IMPALA-1917
+-->
+
+      <xref href="https://issues.cloudera.org/browse/IMPALA-2192" scope="external" format="html">IMPALA-2192</xref>,
+      <xref href="https://issues.cloudera.org/browse/IMPALA-2440" scope="external" format="html">IMPALA-2440</xref>,
+      <xref href="https://issues.cloudera.org/browse/IMPALA-2090" scope="external" format="html">IMPALA-2090</xref>,
+      <xref href="https://issues.cloudera.org/browse/IMPALA-2086" scope="external" format="html">IMPALA-2086</xref>,
+      <xref href="https://issues.cloudera.org/browse/IMPALA-1947" scope="external" format="html">IMPALA-1947</xref>,
+      <xref href="https://issues.cloudera.org/browse/IMPALA-1917" scope="external" format="html">IMPALA-1917</xref>
+      </p>
+      </conbody>
+    </concept>
+
+  </concept>
+
+<!-- All 2.2.x / 5.4.x subsections go under here -->
+
+  <concept rev="2.2.10" id="fixed_issues_2210">
+
+    <title>Issues Fixed in Impala for CDH 5.4.10</title>
+
+    <conbody>
+
+<!--
+      <p>
+        This section lists the most frequently encountered customer issues fixed in Impala for CDH 5.4.10.
+      </p>
+-->
+
+      <p>
+        For the full list of fixed issues for all CDH components, see
+        <xref audience="standalone"
+          href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_54.html#fixed_issues_5410"
+          format="html" scope="external">the CDH 5.4.x release notes</xref><xref
+            audience="integrated" href="cdh_rn_fixed_in_54.xml#fixed_issues_5410"/>.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_22x"/>
+
+      <ul audience="integrated">
+        <li conref="cdh_rn_fixed_in_54.xml#ui_fixed_in_5410/impala_start" conrefend="cdh_rn_fixed_in_54.xml#ui_fixed_in_5410/impala_end"/>
+      </ul>
+
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.2.9" id="fixed_issues_229">
+
+    <title>Issues Fixed in Impala for CDH 5.4.9</title>
+
+    <conbody>
+
+      <p>
+        This section lists the most frequently encountered customer issues fixed in Impala for CDH 5.4.9.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_22x"/>
+
+      <p>
+        For the full list of fixed issues, see
+        <xref audience="standalone"
+          href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_549.html"
+          format="html" scope="external">the CDH 5.4.x release notes</xref><xref
+            audience="integrated" href="cdh_rn_fixed_in_54.xml#fixed_issues_549"/>.
+      </p>
+
+    </conbody>
+
+<!-- List of fixed issues from Juan Yu's spreadsheet: https://docs.google.com/spreadsheets/d/1Ia06xOLKQInKfsJzQfQ6Sf1fO3nYLOpZ-0OWfFNgNSc/edit#gid=1566007471
+
+Asterisks indicate the ones that are considered critical and so get a writeup here.
+
+* 5ff1062 IMPALA-1459: Fix migration/assignment of On-clause predicates inside inline views.
+* bcea8cf IMPALA-2446: Fix wrong predicate assignment in outer joins
+* 8e010af IMPALA-2648/IMPALA-2664: Avoid sending large partition stats objects over thrift
+7e3a789 IMPALA-2260: Adding a large hour interval caused an interval overflow
+* 9a86c94 IMPALA-1675: Avoid overflow when adding large intervals to TIMESTAMPs
+1a83dfc IMPALA-2086, IMPALA-2090: Avoid boost year/month interval logic
+* 1d0b017 IMPALA-1949: Analysis exception when a binary operator contain an IN operator with
+0ae066f IMPALA-2141: UnionNode::GetNext() doesn't check for query errors
+* 31824ad IMPALA-2273: Make MAX_PAGE_HEADER_SIZE configurable
+75497ca IMPALA-2559: Fix check failed: sorter_runs_.back()->is_pinned_
+ddfe66f IMPALA-2265: Sorter was not checking the returned Status of PrepareRead
+* a39a2a5 IMPALA-2357: Fix spilling sorts with var-len slots that are NULL or empty.
+04bcfb9 IMPALA-0000: Fix reporting of custom OOM error messages.
+d9d2a10 IMPALA-0000: Log backtrace on all MEM_LIMIT_EXCEEDED errors
+* 4210021 IMPALA-0000: Work-around IMPALA-2344: Fail query with OOM in case block->Pin() fails
+29d7b42 IMPALA-2286: Fix race between ~BufferedBlockMgr() and BufferedBlockMgr::Creat
+* d87a306 IMPALA-2252: Crash (likely race) tearing down BufferedBlockMgr on query failu
+* a1bb4a6 IMPALA-1746: QueryExecState doesn't check for query cancellation or errors
+* e9ff989 IMPALA-2533: Impala throws IllegalStateException when inserting data into a p
+
+* c22683a IMPALA-1917: Do not register aux equivalence predicates with NULL on either side.
+* c0a7393 IMPALA-2731: Refactor MemPool usage in HBase scan node.
+-->
+
+    <concept id="IMPALA-1917">
+      <!-- Title in Juan's spreadsheet. Actual JIRA title more useful for readers IMO. <title>Do not register aux equivalence predicates with NULL on either side.</title> -->
+      <title>Query return empty result if it contains NullLiteral in inlineview</title>
+      <conbody>
+      <p>
+        If an inline view in a <codeph>FROM</codeph> clause contained a <codeph>NULL</codeph> literal,
+        the result set was empty.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1917" scope="external" format="html">IMPALA-1917</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2731">
+      <!-- Title in Juan's spreadsheet. Actual JIRA title more useful for readers IMO. <title>Refactor MemPool usage in HBase scan node.</title> -->
+      <title>HBase scan node uses 2-4x memory after upgrade to Impala 2.2.8</title>
+      <conbody>
+      <p>
+        Queries involving HBase tables used substantially more memory than in earlier Impala versions.
+        The problem occurred starting in Impala 2.2.8, as a result of the changes for IMPALA-2284.
+        The fix for this issue involves removing a separate memory work area for HBase queries
+        and reusing other memory that was already allocated.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2731" scope="external" format="html">IMPALA-2731</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-1459">
+      <title>Fix migration/assignment of On-clause predicates inside inline views</title>
+      <conbody>
+      <!-- Explanation comes from IMPALA-2665. -->
+      <p>
+        Some combinations of <codeph>ON</codeph> clauses in join queries could result in comparisons
+        being applied at the wrong stage of query processing, leading to incorrect results.
+        Wrong predicate assignment could happen under the following conditions:
+      </p>
+      <ul>
+        <li>
+          The query includes an inline view that contains an outer join.
+        </li>
+        <li>
+          That inline view is joined with another table in the enclosing query block.
+        </li>
+        <li>
+          That join has an <codeph>ON</codeph> clause containing a predicate that
+          only references columns originating from the outer-joined tables inside the inline view.
+        </li>
+      </ul>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1459" scope="external" format="html">IMPALA-1459</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2446">
+      <title>Fix wrong predicate assignment in outer joins</title>
+      <conbody>
+      <p>
+        The join predicate for an <codeph>OUTER JOIN</codeph> clause could be applied at the wrong stage
+        of query processing, leading to incorrect results.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2446" scope="external" format="html">IMPALA-2446</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2648"><!-- Also IMPALA-2664 -->
+      <title>Avoid sending large partition stats objects over thrift</title>
+      <conbody>
+      <p> The <cmdname>catalogd</cmdname> daemon could encounter a serious error when loading the
+          incremental statistics metadata for tables with large numbers of partitions and columns.
+          The problem occurred when the internal representation of metadata for the table exceeded 2
+          GB, for example in a table with 20K partitions and 77 columns. The fix causes a
+            <codeph>COMPUTE INCREMENTAL STATS</codeph> operation to fail if it would produce
+          metadata that exceeded the maximum size. </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2648" scope="external" format="html">IMPALA-2648</xref>,
+      <xref href="https://issues.cloudera.org/browse/IMPALA-2664" scope="external" format="html">IMPALA-2664</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-1675">
+      <title>Avoid overflow when adding large intervals to TIMESTAMPs</title>
+      <conbody>
+      <p> Adding or subtracting a large <codeph>INTERVAL</codeph> value to a
+            <codeph>TIMESTAMP</codeph> value could produce an incorrect result, with the value
+          wrapping instead of returning an out-of-range error. </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1675" scope="external" format="html">IMPALA-1675</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-1949">
+      <title>Analysis exception when a binary operator contains an IN operator with values</title>
+      <conbody>
+      <p>
+        An <codeph>IN</codeph> operator with literal values could cause a statement to fail if used
+        as the argument to a binary operator, such as an equality test for a <codeph>BOOLEAN</codeph> value.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1949" scope="external" format="html">IMPALA-1949</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2273">
+    <!-- To do: this detail would be useful to capture on the Impala + Parquet page too. -->
+      <title>Make MAX_PAGE_HEADER_SIZE configurable</title>
+      <conbody>
+      <p> Impala could fail to access Parquet data files with page headers larger than 8 MB, which
+          could occur, for example, if the minimum or maximum values for a column were long strings.
+          The fix adds a configuration setting <codeph>--max_page_header_size</codeph>, which you
+          can use to increase the Impala size limit to a value higher than 8 MB. </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2273" scope="external" format="html">IMPALA-2273</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2357">
+      <title>Fix spilling sorts with var-len slots that are NULL or empty.</title>
+      <conbody>
+      <p>
+        A query that activated the spill-to-disk mechanism could fail if it contained a sort expression
+        involving certain combinations of fixed-length or variable-length types.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2357" scope="external" format="html">IMPALA-2357</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="block_pin_oom">
+      <title>Work-around IMPALA-2344: Fail query with OOM in case block->Pin() fails</title>
+      <conbody>
+      <p>
+        Some queries that activated the spill-to-disk mechanism could produce a serious error
+        if there was insufficient memory to set up internal work areas. Now those queries
+        produce normal out-of-memory errors instead.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2344" scope="external" format="html">IMPALA-2344</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2252">
+      <title>Crash (likely race) tearing down BufferedBlockMgr on query failure</title>
+      <conbody>
+      <p>
+        A serious error could occur under rare circumstances, due to a race condition while freeing memory during heavily concurrent workloads.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2252" scope="external" format="html">IMPALA-2252</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-1746">
+      <title>QueryExecState doesn't check for query cancellation or errors</title>
+      <conbody>
+      <p>
+        A call to <codeph>SetError()</codeph> in a user-defined function (UDF) would not cause the query to fail as expected.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1746" scope="external" format="html">IMPALA-1746</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2533">
+      <title>Impala throws IllegalStateException when inserting data into a partition while select
+        subquery group by partition columns</title>
+      <conbody>
+      <p>
+        An <codeph>INSERT ... SELECT</codeph> operation into a partitioned table could fail if the <codeph>SELECT</codeph> query
+        included a <codeph>GROUP BY</codeph> clause referring to the partition key columns.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2533" scope="external" format="html">IMPALA-2533</xref></p>
+      </conbody>
+    </concept>
+
+  </concept>
+
+  <concept rev="2.2.8" id="fixed_issues_228">
+
+    <title>Issues Fixed in Impala for CDH 5.4.8</title>
+
+    <conbody>
+
+      <p>
+        This section lists the most frequently encountered customer issues fixed in Impala for CDH 5.4.8.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_22x"/>
+
+      <p>
+        For the full list of fixed issues, see
+        <xref audience="standalone"
+          href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_548.html" format="html" scope="external">the CDH 5.4.x release notes</xref><xref audience="integrated" href="cdh_rn_fixed_in_54.xml#fixed_issues_548"/>.
+      </p>
+
+    </conbody>
+
+<!--
+3e0fee5 IMPALA-2369, IMPALA-2435: Impala crashes when the sorter hits an OOM error
+ef498c2 IMPALA-1136, IMPALA-2161: Skip \u0000 characters when dealing Avro schemas
+8f97d4b IMPALA-2168: Do not try to access streams of repartitioned spilled partition in right-joins
+* 294ac14 Additional checks to catch IMPALA-2168 earlier
+b9b10ec IMPALA-2514: DCHECK on destroying an ExprContext
+96206af IMPALA-2440: Fix old HJ full outer join with no rows
+5e1a79a IMPALA-2366: check fread return code correctly
+02f3e36 IMPALA-2477: Parquet metadata randomly 'appears stale'
+76c1548 IMPALA-2213: make Parquet scanner fail query if the file size metadata is stale
+10ba8ed IMPALA-2249: Avoid allocating StringBuffer > 1GB in ScannerContext::Stream::GetBytesInternal()
+a87eaa0 IMPALA-2270: avoid FnvHash64to32 with empty inputs
+7f6a521 IMPALA-2284: Disallow long (1<<30) strings in group_concat()
+* 9aea5d7 Mistake in schema_constraints by the IMPALA-2130 patch (7c7e69b)
+f8d32a4 IMPALA-2130: Wrong verification of Parquet file version
+9ec5944 IMPALA-2348: The catalog does not close the connection to HMS during table invalidation
+a78c303 IMPALA-2364: Wrong DCHECK in PHJ::ProcessProbeBatch
+e6dec36 IMPALA-2314: LargestSpilledPartition was not checking if partition is closed
+* 56d8bbd Checks for IMPALA-2314 and more accurate calculation of PHJ::LargestSpillPartition
+3ad4e71 IMPALA-2256: Handle joins with right side of high cardinality and zero materialized slots
+4422462 IMPALA-2165: Avoid cardinality 0 in scan nodes of small tables and low selectivity
+1f199ec IMPALA-2292: Change the type of timestamp_col to string in the table no_avro_schema.
+f6b4480 IMPALA-1136: Support loading Avro tables without an explicit Avro schema
+d93fb5a IMPALA-1899: Cleanup handling of Hive's field schema
+
+* = no JIRA assigned with it so it definitely is an internal-only modification.
+-->
+
+    <concept id="IMPALA-1136">
+      <title>Impala is unable to read hive tables created with the "STORED AS AVRO" clause</title>
+      <conbody>
+      <p>Impala could not read Avro tables created in Hive with the <codeph>STORED AS AVRO</codeph> clause.</p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1136" scope="external" format="html">IMPALA-1136</xref>,
+      <xref href="https://issues.cloudera.org/browse/IMPALA-2161" scope="external" format="html">IMPALA-2161</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2213">
+      <title>make Parquet scanner fail query if the file size metadata is stale</title>
+      <conbody>
+      <p>If a Parquet file in HDFS was overwritten by a smaller file, Impala could encounter a serious error.
+      Issuing a <codeph>INVALIDATE METADATA</codeph> statement before a subsequent query would avoid the error.
+      The fix allows Impala to handle such inconsistencies in Parquet file length cleanly regardless of whether the
+      table metadata is up-to-date.</p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2213" scope="external" format="html">IMPALA-2213</xref></p>
+      </conbody>
+    </concept>
+
+     <concept id="IMPALA-2249">
+      <title>Avoid allocating StringBuffer &gt; 1GB in ScannerContext::Stream::GetBytesInternal()</title>
+      <conbody>
+      <p>Impala could encounter a serious error when reading compressed text files larger than 1 GB. The fix causes Impala
+      to issue an error message instead in this case.</p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2249" scope="external" format="html">IMPALA-2249</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2284">
+      <title>Disallow long (1&lt;&lt;30) strings in group_concat()</title>
+      <conbody>
+      <p>A query using the <codeph>group_concat()</codeph> function could encounter a serious error if the returned string value was larger than 1 GB.
+      Now the query fails with an error message in this case.</p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2284" scope="external" format="html">IMPALA-2284</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2270">
+      <title>avoid FnvHash64to32 with empty inputs</title>
+      <conbody>
+      <p>An edge case in the algorithm used to distribute data among nodes could result in uneven distribution of work for some queries,
+      with all data sent to the same node.</p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2270" scope="external" format="html">IMPALA-2270</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2348">
+      <title>The catalog does not close the connection to HMS during table invalidation</title>
+      <conbody>
+      <p>A communication error could occur between Impala and the Hive metastore database, causing Impala operations that update
+      table metadata to fail.</p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2348" scope="external" format="html">IMPALA-2348</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2364-548">
+      <title>Wrong DCHECK in PHJ::ProcessProbeBatch</title>
+      <conbody>
+      <p>Certain queries could encounter a serious error if the spill-to-disk mechanism was activated.</p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2364" scope="external" format="html">IMPALA-2364</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2165-548">
+      <title>Avoid cardinality 0 in scan nodes of small tables and low selectivity</title>
+      <conbody>
+      <p>Impala could generate a suboptimal query plan for some queries involving small tables.</p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2165" scope="external" format="html">IMPALA-2165</xref></p>
+      </conbody>
+    </concept>
+
+  </concept>
+
+  <concept rev="2.2.7" id="fixed_issues_227">
+
+    <title>Issues Fixed in Impala for CDH 5.4.7</title>
+
+    <conbody>
+
+      <p>
+        This section lists the most frequently encountered customer issues fixed in Impala for CDH 5.4.7.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_22x"/>
+
+      <p>
+        For the full list of fixed issues, see
+        <xref audience="standalone"
+          href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_547.html" format="html" scope="external">the CDH 5.4.x release notes</xref><xref audience="integrated" href="cdh_rn_fixed_in_54.xml#fixed_issues_547"/>.
+      </p>
+
+    </conbody>
+<!--
+8736dba IMPALA-1983: Warn if table stats are potentially corrupt.
+a0911fc IMPALA-2266: Pass correct child node in 2nd phase merge aggregation.
+3183878 IMPALA-2216: Set the output smap of an EmptySetNode produced from an empty inline view.
+a73d008 IMPALA-2203: Set an InsertStmt's result exprs from the source statement's result exprs.
+e641759 IMPALA-2088: Fix planning of empty union operands with analytics.
+752b3bd IMPALA-1756: Constant filter expressions are not checked for errors and state cleanup is not done before throwing exception.
+d703959 IMPALA-2239: update misc.test to match the new .test file format
+8dddd60 IMPALA-2089: Retain eq predicates bound by grouping slots with complex grouping exprs.
+ce9fa25 IMPALA-2201: Unconditionally update the partition stats and row count.
+ff5bc02 IMPALA-2199: Row count not set for empty partition when spec is used with compute incremental stats
+3eb2d80 IMPALA-1898: Explicit aliases + ordinals analysis bug
+ba6a8bf IMPALA-1987: Fix TupleIsNullPredicate to return false if no tuples are nullable.
+932d4f8 IMPALA-2178: fix Expr::ComputeResultsLayout() logic
+4ef0bd8 IMPALA-1737: Substitute an InsertStmt's partition key exprs with the root node's smap.
+
+Per Juan Yu, almost all of those are critical issues related to query correctness.
+The non-critical ones are:
+
+752b3bd IMPALA-1756: Constant filter expressions are not checked for errors and state cleanup is not done before throwing exception.
+d703959 IMPALA-2239: update misc.test to match the new .test file format
+ce9fa25 IMPALA-2201: Unconditionally update the partition stats and row count.
+
+-->
+
+    <concept id="IMPALA-1983">
+      <title>Warn if table stats are potentially corrupt.</title>
+      <conbody>
+      <p>
+        Impala warns if it detects a discrepancy in table statistics: a table considered to have zero rows even though there are data files present.
+        In this case, Impala also skips query optimizations that are normally applied to very small tables.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1983" scope="external" format="html">IMPALA-1983:</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2266">
+      <title>Pass correct child node in 2nd phase merge aggregation.</title>
+      <conbody>
+      <p>A query could encounter a serious error if it included a particular combination of aggregate functions and inline views.</p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2266" scope="external" format="html">IMPALA-2266</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2216">
+      <title>Set the output smap of an EmptySetNode produced from an empty inline view.</title>
+      <conbody>
+      <p>A query could encounter a serious error if it included an inline view whose subquery had no <codeph>FROM</codeph> clause.</p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2216" scope="external" format="html">IMPALA-2216</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2203">
+      <title>Set an InsertStmt's result exprs from the source statement's result exprs.</title>
+      <conbody>
+      <p>
+      A <codeph>CREATE TABLE AS SELECT</codeph> or <codeph>INSERT ... SELECT</codeph> statement could produce
+      different results than a <codeph>SELECT</codeph> statement, for queries including a <codeph>FULL JOIN</codeph> clause
+      and including literal values in the select list.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2203" scope="external" format="html">IMPALA-2203</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2088">
+      <title>Fix planning of empty union operands with analytics.</title>
+      <conbody>
+      <p>
+        A query could return incorrect results if it contained a <codeph>UNION</codeph> clause,
+        calls to analytic functions, and a constant expression that evaluated to <codeph>FALSE</codeph>.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2088" scope="external" format="html">IMPALA-2088</xref></p>
+      </conbody>
+    </concept>
+
+<!--
+    <concept id="IMPALA-1756">
+      <title>Constant filter expressions are not checked for errors and state cleanup is not done before throwing exception.</title>
+      <conbody>
+      <p></p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1756" scope="external" format="html">IMPALA-1756</xref></p>
+      <p><b>Severity:</b> High</p>
+      </conbody>
+    </concept>
+-->
+
+<!--
+    <concept id="IMPALA-2239">
+      <title>update misc.test to match the new .test file format</title>
+      <conbody>
+      <p></p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2239" scope="external" format="html">IMPALA-2239</xref></p>
+      <p><b>Severity:</b> High</p>
+      </conbody>
+    </concept>
+-->
+
+    <concept id="IMPALA-2089">
+      <title>Retain eq predicates bound by grouping slots with complex grouping exprs.</title>
+      <conbody>
+      <p>
+        A query containing an <codeph>INNER JOIN</codeph> clause could return undesired rows.
+        Some predicate specified in the <codeph>ON</codeph> clause could be omitted from the filtering operation.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2089" scope="external" format="html">IMPALA-2089</xref></p>
+      </conbody>
+    </concept>
+
+<!--
+    <concept id="IMPALA-2201">
+      <title>Unconditionally update the partition stats and row count.</title>
+      <conbody>
+      <p></p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2201" scope="external" format="html">IMPALA-2201</xref></p>
+      <p><b>Severity:</b> High</p>
+      </conbody>
+    </concept>
+-->
+
+    <concept id="IMPALA-2199">
+      <title>Row count not set for empty partition when spec is used with compute incremental stats</title>
+      <conbody>
+      <p>
+        A <codeph>COMPUTE INCREMENTAL STATS</codeph> statement could leave the row count for an emptyp partition as -1,
+        rather than initializing the row count to 0. The missing statistic value could result in reduced query performance.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2199" scope="external" format="html">IMPALA-2199</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-1898">
+      <title>Explicit aliases + ordinals analysis bug</title>
+      <conbody>
+      <p>
+        A query could encounter a serious error if it included column aliases with the same names as table columns, and used
+        ordinal numbers in an <codeph>ORDER BY</codeph> or <codeph>GROUP BY</codeph> clause.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1898" scope="external" format="html">IMPALA-1898</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-1987">
+      <title>Fix TupleIsNullPredicate to return false if no tuples are nullable.</title>
+      <conbody>
+      <p>
+        A query could return incorrect results if it included an outer join clause, inline views, and calls to functions such as <codeph>coalesce()</codeph>
+        that can generate <codeph>NULL</codeph> values.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1987" scope="external" format="html">IMPALA-1987</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2178">
+      <title>fix Expr::ComputeResultsLayout() logic</title>
+      <conbody>
+      <p>
+        A query could return incorrect results if the table contained multiple <codeph>CHAR</codeph> columns with length of 2 or less,
+        and the query included a <codeph>GROUP BY</codeph> clause that referred to multiple such columns.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2178" scope="external" format="html">IMPALA-2178</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-1737">
+      <title>Substitute an InsertStmt's partition key exprs with the root node's smap.</title>
+      <conbody>
+      <p>
+        An <codeph>INSERT</codeph> statement could encounter a serious error if the <codeph>SELECT</codeph>
+        portion called an analytic function.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1737" scope="external" format="html">IMPALA-1737</xref></p>
+      </conbody>
+    </concept>
+
+  </concept>
+
+  <concept rev="2.2.5" id="fixed_issues_225">
+
+    <title>Issues Fixed in Impala for CDH 5.4.5</title>
+
+    <conbody>
+
+      <p>
+        This section lists the most frequently encountered customer issues fixed in Impala for CDH 5.4.5.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_22x"/>
+
+      <p>
+        For the full list of fixed issues, see
+        <xref audience="standalone"
+          href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_rn_fixed_in_545.html" format="html" scope="external">the CDH 5.4.x release notes</xref><xref audience="integrated" href="cdh_rn_fixed_in_54.xml#fixed_issues_545"/>.
+      </p>
+
+    </conbody>
+
+<!--
+7c01049 IMPALA-1929: Avoiding a DCHECK of NULL hash table in spilled right joins
+c98a556 IMPALA-2136: Bug in PrintTColumnValue caused wrong stats for TINYINT partition cols
+* 4390a5c Making sure that there are no regressions due to IMPALA-2065
+* 33ee7df IMPALA-2133: Properly unescape string value for HBase filters
+* 562b61a IMPALA-1735: ExpandRmReservation only check parent pools with limit
+* 5b09eaa IMPALA-2101: fix NULL ptr deref in BufferedBlockMgr::PinBlock
+c2b8852 IMPALA-2018: Where clause does not propagate to joins inside nested views
+16b79ce IMPALA-2064: Add effective_user() builtin
+99a282b IMPALA-2125: Make UTC to local TimestampValue conversion faster.
+9198985 IMPALA-2065: Workaround IMPALA-1619 in BufferedBlockMgr::ConsumeMemory()
+9ce94eb IMPALA-2048: Set the correct input format when updating partition metadata
+
+* = non-critical to document according to Juan Yu.
+-->
+
+    <concept id="IMPALA-2048">
+      <title>Impala DML/DDL operations corrupt table metadata leading to Hive query failures</title>
+      <!-- Real JIRA title: Class not found: com.cloudera.impala.hive.serde.ParquetInputFormat error -->
+      <conbody>
+      <p>
+        When the Impala <codeph>COMPUTE STATS</codeph> statement was run on a partitioned Parquet table that was created in Hive, the table subsequently became inaccessible in Hive.
+        The table was still accessible to Impala. Regaining access in Hive required a workaround of creating a new table. The error displayed in Hive was:
+      </p>
+<codeblock>Error: Error while compiling statement: FAILED: SemanticException Class not found: com.cloudera.impala.hive.serde.ParquetInputFormat (state=42000,code=40000)</codeblock>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2048" scope="external" format="html">IMPALA-2048</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-1929">
+      <title>Avoiding a DCHECK of NULL hash table in spilled right joins</title>
+      <!-- Real JIRA title: Crash because PHJ::NextSpilledProbeRowBatch() tries to use a NULL hash_tbl -->
+      <conbody>
+      <p>
+        A query could encounter a serious error if it contained a <codeph>RIGHT OUTER</codeph>, <codeph>RIGHT ANTI</codeph>, or <codeph>FULL OUTER</codeph> join clause
+        and approached the memory limit on a host so that the <q>spill to disk</q> mechanism was activated.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-1929" scope="external" format="html">IMPALA-1929</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2136">
+      <title>Bug in PrintTColumnValue caused wrong stats for TINYINT partition cols</title>
+      <!-- Real JIRA title: Partitions with TINYINT partition columns will always have 0 estimated rows after compute stats  -->
+      <conbody>
+      <p>
+        Declaring a partition key column as a <codeph>TINYINT</codeph> caused problems with the <codeph>COMPUTE STATS</codeph> statement.
+        The associated partitions would always have zero estimated rows, leading to potential inefficient query plans.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2136" scope="external" format="html">IMPALA-2136</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2018">
+      <title>Where clause does not propagate to joins inside nested views</title>
+      <!-- Real JIRA title: Where clause does not propagate to joins inside nested views -->
+      <conbody>
+      <p>
+        A query that referred to a view whose query referred to another view containing a join, could return incorrect results.
+        <codeph>WHERE</codeph> clauses for the outermost query were not always applied, causing the result
+        set to include additional rows that should have been filtered out.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2018" scope="external" format="html">IMPALA-2018</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2064">
+      <title>Add effective_user() builtin</title>
+      <!-- Real JIRA title: Add effective_user() builtin -->
+      <conbody>
+      <p>
+        The <codeph>user()</codeph> function returned the name of the logged-in user, which might not be the
+        same as the user name being checked for authorization if, for example, delegation was enabled.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2064" scope="external" format="html">IMPALA-2064</xref></p>
+      <p><b>Resolution:</b> Rather than change the behavior of the <codeph>user()</codeph> function,
+      the fix introduces an additional function <codeph>effective_user()</codeph> that returns the user name that is checked during authorization.</p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2125">
+      <title>Make UTC to local TimestampValue conversion faster.</title>
+      <!-- Real JIRA title: Improve perf when reading timestamps from parquet files written by hive -->
+      <conbody>
+      <p>
+        Query performance was improved substantially for Parquet files containing <codeph>TIMESTAMP</codeph>
+        data written by Hive, when the <codeph>-convert_legacy_hive_parquet_utc_timestamps=true</codeph> setting
+        is in effect.
+      </p>
+      <p><b>Bug:</b> <xref href="https://issues.cloudera.org/browse/IMPALA-2125" scope="external" format="html">IMPALA-2125</xref></p>
+      </conbody>
+    </concept>
+
+    <concept id="IMPALA-2065">
+      <title>Workaround IMPALA-1619 in BufferedBlockMgr::ConsumeMemory()</title>
+      <!-- Real JIRA title: Crash: impala::BufferedBlockMgr::ConsumeMemory

<TRUNCATED>


[19/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_max_block_mgr_memory.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_max_block_mgr_memory.xml b/docs/topics/impala_max_block_mgr_memory.xml
new file mode 100644
index 0000000..3bf8ac8
--- /dev/null
+++ b/docs/topics/impala_max_block_mgr_memory.xml
@@ -0,0 +1,30 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="2.1.0" id="max_block_mgr_memory">
+
+  <title>MAX_BLOCK_MGR_MEMORY</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Memory"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.1.0">
+      <indexterm audience="Cloudera">MAX_BLOCK_MGR_MEMORY query option</indexterm>
+    </p>
+
+    <p></p>
+
+    <p>
+      <b>Default:</b>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/added_in_20"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_max_errors.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_max_errors.xml b/docs/topics/impala_max_errors.xml
new file mode 100644
index 0000000..c6eb971
--- /dev/null
+++ b/docs/topics/impala_max_errors.xml
@@ -0,0 +1,47 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="max_errors">
+
+  <title>MAX_ERRORS Query Option</title>
+  <titlealts audience="PDF"><navtitle>MAX_ERRORS</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Logs"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">MAX_ERRORS query option</indexterm>
+      Maximum number of non-fatal errors for any particular query that are recorded in the Impala log file. For
+      example, if a billion-row table had a non-fatal data error in every row, you could diagnose the problem
+      without all billion errors being logged. Unspecified or 0 indicates the built-in default value of 1000.
+    </p>
+
+    <p>
+      This option only controls how many errors are reported. To specify whether Impala continues or halts when it
+      encounters such errors, use the <codeph>ABORT_ON_ERROR</codeph> option.
+    </p>
+
+    <p>
+      <b>Type:</b> numeric
+    </p>
+
+    <p>
+      <b>Default:</b> 0 (meaning 1000 errors)
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_abort_on_error.xml#abort_on_error"/>,
+      <xref href="impala_logging.xml#logging"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_max_io_buffers.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_max_io_buffers.xml b/docs/topics/impala_max_io_buffers.xml
new file mode 100644
index 0000000..477c833
--- /dev/null
+++ b/docs/topics/impala_max_io_buffers.xml
@@ -0,0 +1,31 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="max_io_buffers">
+
+  <title>MAX_IO_BUFFERS Query Option</title>
+  <titlealts audience="PDF"><navtitle>MAX_IO_BUFFERS</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Deprecated Features"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Deprecated query option. Currently has no effect.
+    </p>
+
+    <p>
+      <b>Type:</b> numeric
+    </p>
+
+    <p>
+      <b>Default:</b> 0
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_max_num_runtime_filters.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_max_num_runtime_filters.xml b/docs/topics/impala_max_num_runtime_filters.xml
new file mode 100644
index 0000000..edf4058
--- /dev/null
+++ b/docs/topics/impala_max_num_runtime_filters.xml
@@ -0,0 +1,61 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="max_num_runtime_filters" rev="2.5.0">
+
+  <title>MAX_NUM_RUNTIME_FILTERS Query Option (<keyword keyref="impala25"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>MAX_NUM_RUNTIME_FILTERS</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.5.0">
+      <indexterm audience="Cloudera">MAX_NUM_RUNTIME_FILTERS query option</indexterm>
+      The <codeph>MAX_NUM_RUNTIME_FILTERS</codeph> query option
+      sets an upper limit on the number of runtime filters that can be produced for each query.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_integer"/>
+
+    <p>
+      <b>Default:</b> 10
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/added_in_250"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      Each runtime filter imposes some memory overhead on the query.
+      Depending on the setting of the <codeph>RUNTIME_BLOOM_FILTER_SIZE</codeph>
+      query option, each filter might consume between 1 and 16 megabytes
+      per plan fragment. There are typically 5 or fewer filters per plan fragment.
+    </p>
+
+    <p>
+      Impala evaluates the effectiveness of each filter, and keeps the
+      ones that eliminate the largest number of partitions or rows.
+      Therefore, this setting can protect against
+      potential problems due to excessive memory overhead for filter production,
+      while still allowing a high level of optimization for suitable queries.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/runtime_filtering_option_caveat"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_runtime_filtering.xml"/>,
+      <!-- <xref href="impala_partitioning.xml#dynamic_partition_pruning"/>, -->
+      <xref href="impala_runtime_bloom_filter_size.xml#runtime_bloom_filter_size"/>,
+      <xref href="impala_runtime_filter_mode.xml#runtime_filter_mode"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_max_scan_range_length.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_max_scan_range_length.xml b/docs/topics/impala_max_scan_range_length.xml
new file mode 100644
index 0000000..a99b92b
--- /dev/null
+++ b/docs/topics/impala_max_scan_range_length.xml
@@ -0,0 +1,55 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="max_scan_range_length">
+
+  <title>MAX_SCAN_RANGE_LENGTH Query Option</title>
+  <titlealts audience="PDF"><navtitle>MAX_SCAN_RANGE_LENGTH</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">MAX_SCAN_RANGE_LENGTH query option</indexterm>
+      Maximum length of the scan range. Interacts with the number of HDFS blocks in the table to determine how many
+      CPU cores across the cluster are involved with the processing for a query. (Each core processes one scan
+      range.)
+    </p>
+
+    <p>
+      Lowering the value can sometimes increase parallelism if you have unused CPU capacity, but a too-small value
+      can limit query performance because each scan range involves extra overhead.
+    </p>
+
+    <p>
+      Only applicable to HDFS tables. Has no effect on Parquet tables. Unspecified or 0 indicates backend default,
+      which is the same as the HDFS block size for each table.
+    </p>
+
+    <p>
+      Although the scan range can be arbitrarily long, Impala internally uses an 8 MB read buffer so that it can
+      query tables with huge block sizes without allocating equivalent blocks of memory.
+    </p>
+
+    <p>
+      <b>Type:</b> numeric
+    </p>
+
+    <p rev="IMPALA-3799 CDH-43746">
+      In <keyword keyref="impala27_full"/> and higher, the argument value can include unit specifiers,
+      such as <codeph>100m</codeph> or <codeph>100mb</codeph>. In previous versions,
+      Impala interpreted such formatted values as 0, leading to query failures.
+    </p>
+
+    <p>
+      <b>Default:</b> 0
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_mem_limit.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_mem_limit.xml b/docs/topics/impala_mem_limit.xml
new file mode 100644
index 0000000..8ed6fbe
--- /dev/null
+++ b/docs/topics/impala_mem_limit.xml
@@ -0,0 +1,212 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="mem_limit">
+
+  <title>MEM_LIMIT Query Option</title>
+  <titlealts audience="PDF"><navtitle>MEM_LIMIT</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Scalability"/>
+      <data name="Category" value="Memory"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">MEM_LIMIT query option</indexterm>
+      When resource management is not enabled, defines the maximum amount of memory a query can allocate on each node.
+      Therefore, the total memory that can be used by a query is the <codeph>MEM_LIMIT</codeph> times the number of nodes.
+    </p>
+
+    <p rev="CDH-32135">
+      There are two levels of memory limit for Impala.
+      The <codeph>-mem_limit</codeph> startup option sets an overall limit for the <cmdname>impalad</cmdname> process
+      (which handles multiple queries concurrently).
+      That limit is typically expressed in terms of a percentage of the RAM available on the host, such as <codeph>-mem_limit=70%</codeph>.
+      The <codeph>MEM_LIMIT</codeph> query option, which you set through <cmdname>impala-shell</cmdname>
+      or the <codeph>SET</codeph> statement in a JDBC or ODBC application, applies to each individual query.
+      The <codeph>MEM_LIMIT</codeph> query option is usually expressed as a fixed size such as <codeph>10gb</codeph>,
+      and must always be less than the <cmdname>impalad</cmdname> memory limit.
+    </p>
+
+    <p rev="CDH-32135">
+      If query processing exceeds the specified memory limit on any node, either the per-query limit or the
+      <cmdname>impalad</cmdname> limit, Impala cancels the query automatically.
+      Memory limits are checked periodically during query processing, so the actual memory in use
+      might briefly exceed the limit without the query being cancelled.
+    </p>
+
+    <p>
+      When resource management is enabled in CDH 5, the mechanism for this option changes. If set, it overrides the
+      automatic memory estimate from Impala. Impala requests this amount of memory from YARN on each node, and the
+      query does not proceed until that much memory is available. The actual memory used by the query could be
+      lower, since some queries use much less memory than others. With resource management, the
+      <codeph>MEM_LIMIT</codeph> setting acts both as a hard limit on the amount of memory a query can use on any
+      node (enforced by YARN) and a guarantee that that much memory will be available on each node while the query
+      is being executed. When resource management is enabled but no <codeph>MEM_LIMIT</codeph> setting is
+      specified, Impala estimates the amount of memory needed on each node for each query, requests that much
+      memory from YARN before starting the query, and then internally sets the <codeph>MEM_LIMIT</codeph> on each
+      node to the requested amount of memory during the query. Thus, if the query takes more memory than was
+      originally estimated, Impala detects that the <codeph>MEM_LIMIT</codeph> is exceeded and cancels the query
+      itself.
+    </p>
+
+    <p>
+      <b>Type:</b> numeric
+    </p>
+
+    <p rev="CDH-32135">
+      <b>Units:</b> A numeric argument represents memory size in bytes; you can also use a suffix of <codeph>m</codeph> or <codeph>mb</codeph>
+      for megabytes, or more commonly <codeph>g</codeph> or <codeph>gb</codeph> for gigabytes. If you specify a value with unrecognized
+      formats, subsequent queries fail with an error.
+    </p>
+
+    <p rev="CDH-32135">
+      <b>Default:</b> 0 (unlimited)
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p rev="CDH-32135">
+      The <codeph>MEM_LIMIT</codeph> setting is primarily useful in a high-concurrency setting,
+      or on a cluster with a workload shared between Impala and other data processing components.
+      You can prevent any query from accidentally using much more memory than expected,
+      which could negatively impact other Impala queries.
+    </p>
+
+    <p rev="CDH-32135">
+      Use the output of the <codeph>SUMMARY</codeph> command in <cmdname>impala-shell</cmdname>
+      to get a report of memory used for each phase of your most heavyweight queries on each node,
+      and then set a <codeph>MEM_LIMIT</codeph> somewhat higher than that.
+      See <xref href="impala_explain_plan.xml#perf_summary"/> for usage information about
+      the <codeph>SUMMARY</codeph> command.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb" rev="CDH-32135"/>
+
+    <p rev="CDH-32135">
+      The following examples show how to set the <codeph>MEM_LIMIT</codeph> query option
+      using a fixed number of bytes, or suffixes representing gigabytes or megabytes.
+    </p>
+
+<codeblock rev="CDH-32135">
+[localhost:21000] > set mem_limit=3000000000;
+MEM_LIMIT set to 3000000000
+[localhost:21000] > select 5;
+Query: select 5
++---+
+| 5 |
++---+
+| 5 |
++---+
+
+[localhost:21000] > set mem_limit=3g;
+MEM_LIMIT set to 3g
+[localhost:21000] > select 5;
+Query: select 5
++---+
+| 5 |
++---+
+| 5 |
++---+
+
+[localhost:21000] > set mem_limit=3gb;
+MEM_LIMIT set to 3gb
+[localhost:21000] > select 5;
++---+
+| 5 |
++---+
+| 5 |
++---+
+
+[localhost:21000] > set mem_limit=3m;
+MEM_LIMIT set to 3m
+[localhost:21000] > select 5;
++---+
+| 5 |
++---+
+| 5 |
++---+
+[localhost:21000] > set mem_limit=3mb;
+MEM_LIMIT set to 3mb
+[nightly55-2.vpc.cloudera.com:21000] > select 5;
++---+
+| 5 |
++---+
+</codeblock>
+
+    <p rev="CDH-32135">
+      The following examples show how unrecognized <codeph>MEM_LIMIT</codeph>
+      values lead to errors for subsequent queries.
+    </p>
+
+<codeblock rev="CDH-32135">
+[localhost:21000] > set mem_limit=3tb;
+MEM_LIMIT set to 3tb
+[localhost:21000] > select 5;
+ERROR: Failed to parse query memory limit from '3tb'.
+
+[localhost:21000] > set mem_limit=xyz;
+MEM_LIMIT set to xyz
+[localhost:21000] > select 5;
+Query: select 5
+ERROR: Failed to parse query memory limit from 'xyz'.
+</codeblock>
+
+    <p rev="CDH-32135">
+      The following examples shows the automatic query cancellation
+      when the <codeph>MEM_LIMIT</codeph> value is exceeded
+      on any host involved in the Impala query. First it runs a
+      successful query and checks the largest amount of memory
+      used on any node for any stage of the query.
+      Then it sets an artificially low <codeph>MEM_LIMIT</codeph>
+      setting so that the same query cannot run.
+    </p>
+
+<codeblock rev="CDH-32135">
+[localhost:21000] > select count(*) from customer;
+Query: select count(*) from customer
++----------+
+| count(*) |
++----------+
+| 150000   |
++----------+
+
+[localhost:21000] > select count(distinct c_name) from customer;
+Query: select count(distinct c_name) from customer
++------------------------+
+| count(distinct c_name) |
++------------------------+
+| 150000                 |
++------------------------+
+
+[localhost:21000] > summary;
++--------------+--------+----------+----------+---------+------------+----------+---------------+---------------+
+| Operator     | #Hosts | Avg Time | Max Time | #Rows   | Est. #Rows | Peak Mem | Est. Peak Mem | Detail        |
++--------------+--------+----------+----------+---------+------------+----------+---------------+---------------+
+| 06:AGGREGATE | 1      | 230.00ms | 230.00ms | 1       | 1          | 16.00 KB | -1 B          | FINALIZE      |
+| 05:EXCHANGE  | 1      | 43.44us  | 43.44us  | 1       | 1          | 0 B      | -1 B          | UNPARTITIONED |
+| 02:AGGREGATE | 1      | 227.14ms | 227.14ms | 1       | 1          | 12.00 KB | 10.00 MB      |               |
+| 04:AGGREGATE | 1      | 126.27ms | 126.27ms | 150.00K | 150.00K    | 15.17 MB | 10.00 MB      |               |
+| 03:EXCHANGE  | 1      | 44.07ms  | 44.07ms  | 150.00K | 150.00K    | 0 B      | 0 B           | HASH(c_name)  |
+<b>| 01:AGGREGATE | 1      | 361.94ms | 361.94ms | 150.00K | 150.00K    | 23.04 MB | 10.00 MB      |               |</b>
+| 00:SCAN HDFS | 1      | 43.64ms  | 43.64ms  | 150.00K | 150.00K    | 24.19 MB | 64.00 MB      | tpch.customer |
++--------------+--------+----------+----------+---------+------------+----------+---------------+---------------+
+
+[localhost:21000] > set mem_limit=15mb;
+MEM_LIMIT set to 15mb
+[localhost:21000] > select count(distinct c_name) from customer;
+Query: select count(distinct c_name) from customer
+ERROR:
+Memory limit exceeded
+Query did not have enough memory to get the minimum required buffers in the block manager.
+</codeblock>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_min.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_min.xml b/docs/topics/impala_min.xml
new file mode 100644
index 0000000..047b4ef
--- /dev/null
+++ b/docs/topics/impala_min.xml
@@ -0,0 +1,197 @@
+<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="min">
+
+  <title>MIN Function</title>
+  <titlealts audience="PDF"><navtitle>MIN</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="Analytic Functions"/>
+      <data name="Category" value="Aggregate Functions"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">min() function</indexterm>
+      An aggregate function that returns the minimum value from a set of numbers. Opposite of the
+      <codeph>MAX</codeph> function. Its single argument can be numeric column, or the numeric result of a function
+      or expression applied to the column value. Rows with a <codeph>NULL</codeph> value for the specified column
+      are ignored. If the table is empty, or all the values supplied to <codeph>MIN</codeph> are
+      <codeph>NULL</codeph>, <codeph>MIN</codeph> returns <codeph>NULL</codeph>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>MIN([DISTINCT | ALL] <varname>expression</varname>) [OVER (<varname>analytic_clause</varname>)]</codeblock>
+
+    <p>
+      When the query contains a <codeph>GROUP BY</codeph> clause, returns one value for each combination of
+      grouping values.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_sliding_window"/>
+
+    <p conref="../shared/impala_common.xml#common/return_type_same_except_string"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/partition_key_optimization"/>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_aggregation_explanation"/>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_aggregation_example"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+<codeblock>-- Find the smallest value for this column in the table.
+select min(c1) from t1;
+-- Find the smallest value for this column from a subset of the table.
+select min(c1) from t1 where month = 'January' and year = '2013';
+-- Find the smallest value from a set of numeric function results.
+select min(length(s)) from t1;
+-- Can also be used in combination with DISTINCT and/or GROUP BY.
+-- Return more than one result.
+select month, year, min(purchase_price) from store_stats group by month, year;
+-- Filter the input to eliminate duplicates before performing the calculation.
+select min(distinct x) from t1;
+</codeblock>
+
+    <p rev="2.0.0">
+      The following examples show how to use <codeph>MIN()</codeph> in an analytic context. They use a table
+      containing integers from 1 to 10. Notice how the <codeph>MIN()</codeph> is reported for each input value, as
+      opposed to the <codeph>GROUP BY</codeph> clause which condenses the result set.
+<codeblock>select x, property, min(x) over (partition by property) as min from int_t where property in ('odd','even');
++----+----------+-----+
+| x  | property | min |
++----+----------+-----+
+| 2  | even     | 2   |
+| 4  | even     | 2   |
+| 6  | even     | 2   |
+| 8  | even     | 2   |
+| 10 | even     | 2   |
+| 1  | odd      | 1   |
+| 3  | odd      | 1   |
+| 5  | odd      | 1   |
+| 7  | odd      | 1   |
+| 9  | odd      | 1   |
++----+----------+-----+
+</codeblock>
+
+Adding an <codeph>ORDER BY</codeph> clause lets you experiment with results that are cumulative or apply to a moving
+set of rows (the <q>window</q>). The following examples use <codeph>MIN()</codeph> in an analytic context
+(that is, with an <codeph>OVER()</codeph> clause) to display the smallest value of <codeph>X</codeph>
+encountered up to each row in the result set. The examples use two columns in the <codeph>ORDER BY</codeph>
+clause to produce a sequence of values that rises and falls, to illustrate how the <codeph>MIN()</codeph>
+result only decreases or stays the same throughout each partition within the result set.
+The basic <codeph>ORDER BY x</codeph> clause implicitly
+activates a window clause of <codeph>RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW</codeph>,
+which is effectively the same as <codeph>ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW</codeph>,
+therefore all of these examples produce the same results:
+
+<codeblock>select x, property, min(x) <b>over (order by property, x desc)</b> as 'minimum to this point'
+  from int_t where property in ('prime','square');
++---+----------+-----------------------+
+| x | property | minimum to this point |
++---+----------+-----------------------+
+| 7 | prime    | 7                     |
+| 5 | prime    | 5                     |
+| 3 | prime    | 3                     |
+| 2 | prime    | 2                     |
+| 9 | square   | 2                     |
+| 4 | square   | 2                     |
+| 1 | square   | 1                     |
++---+----------+-----------------------+
+
+select x, property,
+  min(x) over
+  (
+    <b>order by property, x desc</b>
+    <b>range between unbounded preceding and current row</b>
+  ) as 'minimum to this point'
+from int_t where property in ('prime','square');
++---+----------+-----------------------+
+| x | property | minimum to this point |
++---+----------+-----------------------+
+| 7 | prime    | 7                     |
+| 5 | prime    | 5                     |
+| 3 | prime    | 3                     |
+| 2 | prime    | 2                     |
+| 9 | square   | 2                     |
+| 4 | square   | 2                     |
+| 1 | square   | 1                     |
++---+----------+-----------------------+
+
+select x, property,
+  min(x) over
+  (
+    <b>order by property, x desc</b>
+    <b>rows between unbounded preceding and current row</b>
+  ) as 'minimum to this point'
+from int_t where property in ('prime','square');
++---+----------+-----------------------+
+| x | property | minimum to this point |
++---+----------+-----------------------+
+| 7 | prime    | 7                     |
+| 5 | prime    | 5                     |
+| 3 | prime    | 3                     |
+| 2 | prime    | 2                     |
+| 9 | square   | 2                     |
+| 4 | square   | 2                     |
+| 1 | square   | 1                     |
++---+----------+-----------------------+
+</codeblock>
+
+The following examples show how to construct a moving window, with a running minimum taking into account all rows before
+and 1 row after the current row.
+Because of a restriction in the Impala <codeph>RANGE</codeph> syntax, this type of
+moving window is possible with the <codeph>ROWS BETWEEN</codeph> clause but not the <codeph>RANGE BETWEEN</codeph> clause.
+Because of an extra Impala restriction on the <codeph>MAX()</codeph> and <codeph>MIN()</codeph> functions in an
+analytic context, the lower bound must be <codeph>UNBOUNDED PRECEDING</codeph>.
+<codeblock>select x, property,
+  min(x) over
+  (
+    <b>order by property, x desc</b>
+    <b>rows between unbounded preceding and 1 following</b>
+  ) as 'local minimum'
+from int_t where property in ('prime','square');
++---+----------+---------------+
+| x | property | local minimum |
++---+----------+---------------+
+| 7 | prime    | 5             |
+| 5 | prime    | 3             |
+| 3 | prime    | 2             |
+| 2 | prime    | 2             |
+| 9 | square   | 2             |
+| 4 | square   | 1             |
+| 1 | square   | 1             |
++---+----------+---------------+
+
+-- Doesn't work because of syntax restriction on RANGE clause.
+select x, property,
+  min(x) over
+  (
+    <b>order by property, x desc</b>
+    <b>range between unbounded preceding and 1 following</b>
+  ) as 'local minimum'
+from int_t where property in ('prime','square');
+ERROR: AnalysisException: RANGE is only supported with both the lower and upper bounds UNBOUNDED or one UNBOUNDED and the other CURRENT ROW.
+</codeblock>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      <xref href="impala_analytic_functions.xml#analytic_functions"/>, <xref href="impala_max.xml#max"/>,
+      <xref href="impala_avg.xml#avg"/>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_misc_functions.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_misc_functions.xml b/docs/topics/impala_misc_functions.xml
new file mode 100644
index 0000000..196f3c2
--- /dev/null
+++ b/docs/topics/impala_misc_functions.xml
@@ -0,0 +1,196 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="misc_functions">
+
+  <title>Impala Miscellaneous Functions</title>
+  <titlealts audience="PDF"><navtitle>Miscellaneous Functions</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Querying"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Impala supports the following utility functions that do not operate on a particular column or data type:
+    </p>
+
+    <dl>
+      <dlentry rev="1.3.0" id="current_database">
+
+        <dt>
+          <codeph>current_database()</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">current_database() function</indexterm>
+          <b>Purpose:</b> Returns the database that the session is currently using, either <codeph>default</codeph>
+          if no database has been selected, or whatever database the session switched to through a
+          <codeph>USE</codeph> statement or the <cmdname>impalad</cmdname><codeph>-d</codeph> option.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="5.4.5" id="effective_user">
+
+        <dt>
+          <codeph>effective_user()</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">effective_user() function</indexterm>
+          <b>Purpose:</b> Typically returns the same value as <codeph>user()</codeph>,
+          except if delegation is enabled, in which case it returns the ID of the delegated user.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+          <p>
+            <b>Added in:</b> CDH 5.4.5
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.3.0" id="pid">
+
+        <dt>
+          <codeph>pid()</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">pid() function</indexterm>
+          <b>Purpose:</b> Returns the process ID of the <cmdname>impalad</cmdname> daemon that the session is
+          connected to. You can use it during low-level debugging, to issue Linux commands that trace, show the
+          arguments, and so on the <cmdname>impalad</cmdname> process.
+          <p>
+            <b>Return type:</b> <codeph>int</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry audience="Cloudera" id="sleep">
+
+        <dt>
+          <codeph>sleep(int ms)</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">sleep() function</indexterm>
+          <b>Purpose:</b> Pauses the query for a specified number of milliseconds. For slowing down queries with
+          small result sets enough to monitor runtime execution, memory usage, or other factors that otherwise
+          would be difficult to capture during the brief interval of query execution. When used in the
+          <codeph>SELECT</codeph> list, it is called once for each row in the result set; adjust the number of
+          milliseconds accordingly. For example, a query <codeph>SELECT *, sleep(5) FROM
+          table_with_1000_rows</codeph> would take at least 5 seconds to complete (5 milliseconds * 1000 rows in
+          result set). To avoid an excessive number of concurrent queries, use this function for troubleshooting on
+          test and development systems, not for production queries.
+          <p>
+            <b>Return type:</b> N/A
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="1.1" id="user">
+
+        <dt>
+          <codeph>user()</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">user() function</indexterm>
+          <b>Purpose:</b> Returns the username of the Linux user who is connected to the <cmdname>impalad</cmdname>
+          daemon. Typically called a single time, in a query without any <codeph>FROM</codeph> clause, to
+          understand how authorization settings apply in a security context; once you know the logged-in username,
+          you can check which groups that user belongs to, and from the list of groups you can check which roles
+          are available to those groups through the authorization policy file.
+          <p conref="../shared/impala_common.xml#common/user_kerberized"/>
+          <p>
+            When delegation is enabled, consider calling the <codeph>effective_user()</codeph> function instead.
+          </p>
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+        </dd>
+
+      </dlentry>
+
+      <dlentry rev="2.5.0 IMPALA-1477" id="uuid">
+
+        <dt>
+          <codeph>uuid()</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">uuid() function</indexterm>
+          <b>Purpose:</b> Returns a <xref href="https://en.wikipedia.org/wiki/Universally_unique_identifier" scope="external" format="html">universal unique identifier</xref>, a 128-bit value encoded as a string with groups of hexadecimal digits separated by dashes.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph>
+          </p>
+          <p conref="../shared/impala_common.xml#common/added_in_250"/>
+          <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+          <p>
+            Ascending numeric sequences of type <codeph>BIGINT</codeph> are often used
+            as identifiers within a table, and as join keys across multiple tables.
+            The <codeph>uuid()</codeph> value is a convenient alternative that does not
+            require storing or querying the highest sequence number. For example, you
+            can use it to quickly construct new unique identifiers during a data import job,
+            or to combine data from different tables without the likelihood of ID collisions.
+          </p>
+          <p conref="../shared/impala_common.xml#common/example_blurb"/>
+<codeblock>
+-- Each call to uuid() produces a new arbitrary value.
+select uuid();
++--------------------------------------+
+| uuid()                               |
++--------------------------------------+
+| c7013e25-1455-457f-bf74-a2046e58caea |
++--------------------------------------+
+
+-- If you get a UUID for each row of a result set, you can use it as a
+-- unique identifier within a table, or even a unique ID across tables.
+select uuid() from four_row_table;
++--------------------------------------+
+| uuid()                               |
++--------------------------------------+
+| 51d3c540-85e5-4cb9-9110-604e53999e2e |
+| 0bb40071-92f6-4a59-a6a4-60d46e9703e2 |
+| 5e9d7c36-9842-4a96-862d-c13cd0457c02 |
+| cae29095-0cc0-4053-a5ea-7fcd3c780861 |
++--------------------------------------+
+</codeblock>
+        </dd>
+
+      </dlentry>
+
+      <dlentry id="version">
+
+        <dt>
+          <codeph>version()</codeph>
+        </dt>
+
+        <dd>
+          <indexterm audience="Cloudera">version() function</indexterm>
+          <b>Purpose:</b> Returns information such as the precise version number and build date for the
+          <codeph>impalad</codeph> daemon that you are currently connected to. Typically used to confirm that you
+          are connected to the expected level of Impala to use a particular feature, or to connect to several nodes
+          and confirm they are all running the same level of <cmdname>impalad</cmdname>.
+          <p>
+            <b>Return type:</b> <codeph>string</codeph> (with one or more embedded newlines)
+          </p>
+        </dd>
+
+      </dlentry>
+    </dl>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_mixed_security.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_mixed_security.xml b/docs/topics/impala_mixed_security.xml
new file mode 100644
index 0000000..b9e6933
--- /dev/null
+++ b/docs/topics/impala_mixed_security.xml
@@ -0,0 +1,46 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="mixed_security">
+
+  <title>Using Multiple Authentication Methods with Impala</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Authentication"/>
+      <data name="Category" value="Kerberos"/>
+      <data name="Category" value="LDAP"/>
+      <data name="Category" value="Administrators"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Impala 2.0 and later automatically handles both Kerberos and LDAP authentication. Each
+      <cmdname>impalad</cmdname> daemon can accept both Kerberos and LDAP requests through the same port. No
+      special actions need to be taken if some users authenticate through Kerberos and some through LDAP.
+    </p>
+
+    <p>
+      Prior to Impala 2.0, you had to configure each <cmdname>impalad</cmdname> to listen on a specific port
+      depending on the kind of authentication, then configure your network load balancer to forward each kind of
+      request to a DataNode that was set up with the appropriate authentication type. Once the initial request was
+      made using either Kerberos or LDAP authentication, Impala automatically handled the process of coordinating
+      the work across multiple nodes and transmitting intermediate results back to the coordinator node.
+    </p>
+
+<!--
+    <p>
+    This technique is most suitable for larger clusters, where
+    you are already using load balancing software for high availability.
+    You configure Impala to run on a different port on the nodes configured for LDAP.
+    Then you configure the load balancing software to forward Kerberos
+    connection requests to nodes using the default port, and LDAP connection requests
+    to nodes using an alternative port for LDAP.
+    Consult the documentation for your load balancing software for how to
+    configure that type of forwarding.
+    </p>
+-->
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_ndv.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_ndv.xml b/docs/topics/impala_ndv.xml
new file mode 100644
index 0000000..fcdae77
--- /dev/null
+++ b/docs/topics/impala_ndv.xml
@@ -0,0 +1,135 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.2.1" id="ndv">
+
+  <title>NDV Function</title>
+  <titlealts audience="PDF"><navtitle>NDV</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Impala Functions"/>
+      <data name="Category" value="Aggregate Functions"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">ndv() function</indexterm>
+      An aggregate function that returns an approximate value similar to the result of <codeph>COUNT(DISTINCT
+      <varname>col</varname>)</codeph>, the <q>number of distinct values</q>. It is much faster than the
+      combination of <codeph>COUNT</codeph> and <codeph>DISTINCT</codeph>, and uses a constant amount of memory and
+      thus is less memory-intensive for columns with high cardinality.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>NDV([DISTINCT | ALL] <varname>expression</varname>)</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p rev="1.2.2">
+      This is the mechanism used internally by the <codeph>COMPUTE STATS</codeph> statement for computing the
+      number of distinct values in a column.
+    </p>
+
+    <p>
+      Because this number is an estimate, it might not reflect the precise number of different values in the
+      column, especially if the cardinality is very low or very high. If the estimated number is higher than the
+      number of rows in the table, Impala adjusts the value internally during query planning.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/former_odd_return_type_string"/>
+
+<!-- <p conref="../shared/impala_common.xml#common/restrictions_sliding_window"/> -->
+
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_aggregation_explanation"/>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_aggregation_example"/>
+
+    <p conref="../shared/impala_common.xml#common/restrictions_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/analytic_not_allowed_caveat"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      The following example queries a billion-row table to illustrate the relative performance of
+      <codeph>COUNT(DISTINCT)</codeph> and <codeph>NDV()</codeph>. It shows how <codeph>COUNT(DISTINCT)</codeph>
+      gives a precise answer, but is inefficient for large-scale data where an approximate result is sufficient.
+      The <codeph>NDV()</codeph> function gives an approximate result but is much faster.
+    </p>
+
+<codeblock>select count(distinct col1) from sample_data;
++---------------------+
+| count(distinct col1)|
++---------------------+
+| 100000              |
++---------------------+
+Fetched 1 row(s) in 20.13s
+
+select cast(ndv(col1) as bigint) as col1 from sample_data;
++----------+
+| col1     |
++----------+
+| 139017   |
++----------+
+Fetched 1 row(s) in 8.91s
+</codeblock>
+
+    <p>
+      The following example shows how you can code multiple <codeph>NDV()</codeph> calls in a single query, to
+      easily learn which columns have substantially more or fewer distinct values. This technique is faster than
+      running a sequence of queries with <codeph>COUNT(DISTINCT)</codeph> calls.
+    </p>
+
+<codeblock>select cast(ndv(col1) as bigint) as col1, cast(ndv(col2) as bigint) as col2,
+    cast(ndv(col3) as bigint) as col3, cast(ndv(col4) as bigint) as col4
+  from sample_data;
++----------+-----------+------------+-----------+
+| col1     | col2      | col3       | col4      |
++----------+-----------+------------+-----------+
+| 139017   | 282       | 46         | 145636240 |
++----------+-----------+------------+-----------+
+Fetched 1 row(s) in 34.97s
+
+select count(distinct col1) from sample_data;
++---------------------+
+| count(distinct col1)|
++---------------------+
+| 100000              |
++---------------------+
+Fetched 1 row(s) in 20.13s
+
+select count(distinct col2) from sample_data;
++----------------------+
+| count(distinct col2) |
++----------------------+
+| 278                  |
++----------------------+
+Fetched 1 row(s) in 20.09s
+
+select count(distinct col3) from sample_data;
++-----------------------+
+| count(distinct col3)  |
++-----------------------+
+| 46                    |
++-----------------------+
+Fetched 1 row(s) in 19.12s
+
+select count(distinct col4) from sample_data;
++----------------------+
+| count(distinct col4) |
++----------------------+
+| 147135880            |
++----------------------+
+Fetched 1 row(s) in 266.95s
+</codeblock>
+  </conbody>
+</concept>



[25/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_identifiers.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_identifiers.xml b/docs/topics/impala_identifiers.xml
new file mode 100644
index 0000000..7a947b3
--- /dev/null
+++ b/docs/topics/impala_identifiers.xml
@@ -0,0 +1,114 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="identifiers">
+
+  <title>Overview of Impala Identifiers</title>
+  <titlealts audience="PDF"><navtitle>Identifiers</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Databases"/>
+      <data name="Category" value="Tables"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Identifiers are the names of databases, tables, or columns that you specify in a SQL statement. The rules for
+      identifiers govern what names you can give to things you create, the notation for referring to names
+      containing unusual characters, and other aspects such as case sensitivity.
+    </p>
+
+    <ul>
+      <li>
+        <p>
+        The minimum length of an identifier is 1 character.
+        </p>
+      </li>
+
+      <li>
+        <p>
+        The maximum length of an identifier is currently 128 characters, enforced by the metastore database.
+        </p>
+      </li>
+
+      <li>
+        <p>
+        An identifier must start with an alphabetic character. The remainder can contain any combination of
+        alphanumeric characters and underscores. Quoting the identifier with backticks has no effect on the allowed
+        characters in the name.
+        </p>
+      </li>
+
+      <li>
+        <p>
+        An identifier can contain only ASCII characters.
+        </p>
+      </li>
+
+      <li>
+        <p>
+        To use an identifier name that matches one of the Impala reserved keywords (listed in
+        <xref href="impala_reserved_words.xml#reserved_words"/>), surround the identifier with <codeph>``</codeph>
+        characters (backticks). Quote the reserved word even if it is part of a fully qualified name.
+        The following example shows how a reserved word can be used as a column name if it is quoted
+        with backticks in the <codeph>CREATE TABLE</codeph> statement, and how the column name
+        must also be quoted with backticks in a query:
+        </p>
+<codeblock>[localhost:21000] > create table reserved (`data` string);
+
+[localhost:21000] > select data from reserved;
+ERROR: AnalysisException: Syntax error in line 1:
+select data from reserved
+       ^
+Encountered: DATA
+Expected: ALL, CASE, CAST, DISTINCT, EXISTS, FALSE, IF, INTERVAL, NOT, NULL, STRAIGHT_JOIN, TRUE, IDENTIFIER
+CAUSED BY: Exception: Syntax error
+
+[localhost:21000] > select reserved.data from reserved;
+ERROR: AnalysisException: Syntax error in line 1:
+select reserved.data from reserved
+                ^
+Encountered: DATA
+Expected: IDENTIFIER
+CAUSED BY: Exception: Syntax error
+
+[localhost:21000] > select reserved.`data` from reserved;
+
+[localhost:21000] >
+</codeblock>
+
+        <note type="important">
+          Because the list of reserved words grows over time as new SQL syntax is added,
+          consider adopting coding conventions (especially for any automated scripts
+          or in packaged applications) to always quote all identifiers with backticks.
+          Quoting all identifiers protects your SQL from compatibility issues if
+          new reserved words are added in later releases.
+        </note>
+
+      </li>
+
+      <li>
+        <p>
+        Impala identifiers are always case-insensitive. That is, tables named <codeph>t1</codeph> and
+        <codeph>T1</codeph> always refer to the same table, regardless of quote characters. Internally, Impala
+        always folds all specified table and column names to lowercase. This is why the column headers in query
+        output are always displayed in lowercase.
+        </p>
+      </li>
+    </ul>
+
+    <p>
+      See <xref href="impala_aliases.xml#aliases"/> for how to define shorter or easier-to-remember aliases if the
+      original names are long or cryptic identifiers.
+      <ph conref="../shared/impala_common.xml#common/aliases_vs_identifiers"/>
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/views_vs_identifiers"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_impala_shell.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_impala_shell.xml b/docs/topics/impala_impala_shell.xml
new file mode 100644
index 0000000..afbfcaa
--- /dev/null
+++ b/docs/topics/impala_impala_shell.xml
@@ -0,0 +1,101 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="impala_shell">
+
+  <title>Using the Impala Shell (impala-shell Command)</title>
+  <titlealts audience="PDF"><navtitle>The Impala Shell</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="impala-shell"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Stub Pages"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">impala-shell</indexterm>
+      You can use the Impala shell tool (<codeph>impala-shell</codeph>) to set up databases and tables, insert
+      data, and issue queries. For ad hoc queries and exploration, you can submit SQL statements in an interactive
+      session. To automate your work, you can specify command-line options to process a single statement or a
+      script file. The <cmdname>impala-shell</cmdname> interpreter accepts all the same SQL statements listed in
+      <xref href="impala_langref_sql.xml#langref_sql"/>, plus some shell-only commands that you can use for tuning
+      performance and diagnosing problems.
+    </p>
+
+    <p>
+      The <codeph>impala-shell</codeph> command fits into the familiar Unix toolchain:
+    </p>
+
+    <ul>
+      <li>
+        The <codeph>-q</codeph> option lets you issue a single query from the command line, without starting the
+        interactive interpreter. You could use this option to run <codeph>impala-shell</codeph> from inside a shell
+        script or with the command invocation syntax from a Python, Perl, or other kind of script.
+      </li>
+
+      <li>
+        The <codeph>-f</codeph> option lets you process a file containing multiple SQL statements,
+        such as a set of reports or DDL statements to create a group of tables and views.
+      </li>
+
+      <li rev="2.5.0 IMPALA-2179">
+        The <codeph>--var</codeph> option lets you pass substitution variables to the statements that
+        are executed by that <cmdname>impala-shell</cmdname> session, for example the statements
+        in a script file processed by the <codeph>-f</codeph> option. You encode the substitution variable
+        on the command line using the notation
+        <codeph>--var=<varname>variable_name</varname>=<varname>value</varname></codeph>.
+        Within a SQL statement, you substitute the value by using the notation <codeph>${var:<varname>variable_name</varname>}</codeph>.
+        This feature is available in <keyword keyref="impala25_full"/> and higher.
+      </li>
+
+      <li>
+        The <codeph>-o</codeph> option lets you save query output to a file.
+      </li>
+
+      <li>
+        The <codeph>-B</codeph> option turns off pretty-printing, so that you can produce comma-separated,
+        tab-separated, or other delimited text files as output. (Use the <codeph>--output_delimiter</codeph> option
+        to choose the delimiter character; the default is the tab character.)
+      </li>
+
+      <li>
+        In non-interactive mode, query output is printed to <codeph>stdout</codeph> or to the file specified by the
+        <codeph>-o</codeph> option, while incidental output is printed to <codeph>stderr</codeph>, so that you can
+        process just the query output as part of a Unix pipeline.
+      </li>
+
+      <li>
+        In interactive mode, <codeph>impala-shell</codeph> uses the <codeph>readline</codeph> facility to recall
+        and edit previous commands.
+      </li>
+    </ul>
+
+    <p>
+      For information on installing the Impala shell, see <xref href="impala_install.xml#install"/>. In Cloudera
+      Manager 4.1 and higher, Cloudera Manager installs <cmdname>impala-shell</cmdname> automatically. You might
+      install <cmdname>impala-shell</cmdname> manually on other systems not managed by Cloudera Manager, so that
+      you can issue queries from client systems that are not also running the Impala daemon or other Apache Hadoop
+      components.
+    </p>
+
+    <p>
+      For information about establishing a connection to a DataNode running the <codeph>impalad</codeph> daemon
+      through the <codeph>impala-shell</codeph> command, see <xref href="impala_connecting.xml#connecting"/>.
+    </p>
+
+    <p>
+      For a list of the <codeph>impala-shell</codeph> command-line options, see
+      <xref href="impala_shell_options.xml#shell_options"/>. For reference information about the
+      <codeph>impala-shell</codeph> interactive commands, see
+      <xref href="impala_shell_commands.xml#shell_commands"/>.
+    </p>
+
+    <p outputclass="toc"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_incompatible_changes.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_incompatible_changes.xml b/docs/topics/impala_incompatible_changes.xml
new file mode 100644
index 0000000..88a306f
--- /dev/null
+++ b/docs/topics/impala_incompatible_changes.xml
@@ -0,0 +1,1519 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="ver" id="incompatible_changes">
+
+  <title><ph audience="standalone">Incompatible Changes and Limitations in Apache Impala (incubating)</ph><ph audience="integrated">Apache Impala (incubating) Incompatible Changes and Limitations</ph></title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Release Notes"/>
+      <data name="Category" value="Incompatible Changes"/>
+      <data name="Category" value="Limitations"/>
+      <data name="Category" value="Upgrading"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      The Impala version covered by this documentation library contains the following incompatible changes. These
+      are things such as file format changes, removed features, or changes to implementation, default
+      configuration, dependencies, or prerequisites that could cause issues during or after an Impala upgrade.
+    </p>
+
+    <p>
+      Even added SQL statements or clauses can produce incompatibilities, if you have databases, tables, or columns
+      whose names conflict with the new keywords. <ph audience="PDF">See
+      <xref href="impala_reserved_words.xml#reserved_words"/> for the set of reserved words for the current
+      release, and the quoting techniques to avoid name conflicts.</ph>
+    </p>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept rev="2.7.0" id="incompatible_changes_27x">
+
+    <title>Incompatible Changes Introduced in Impala for CDH 5.9.x / Impala 2.7.x</title>
+
+    <conbody>
+      <ul>
+        <li>
+          <p rev="IMPALA-1731 IMPALA-3868 CDH-43734">
+            Bug fixes related to parsing of floating-point values (IMPALA-1731 and IMPALA-3868) can change
+            the results of casting strings that represent invalid floating-point values.
+            For example, formerly a string value beginning or ending with <codeph>inf</codeph>,
+            such as <codeph>1.23inf</codeph> or <codeph>infinite</codeph>, now are converted to <codeph>NULL</codeph>
+            when interpreted as a floating-point value.
+            Formerly, they were interpreted as the special <q>infinity</q> value when converting from string to floating-point.
+            Similarly, now only the string <codeph>NaN</codeph> (case-sensitive) is interpreted as the special <q>not a number</q>
+            value. String values containing multiple dots, such as <codeph>3..141</codeph> or <codeph>3.1.4.1</codeph>,
+            are now interpreted as <codeph>NULL</codeph> rather than being converted to valid floating-point values.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-4372">
+            The column types shown in the <codeph>DESCRIBE FORMATTED</codeph> output are in uppercase, where formerly they
+            were in lowercase. This is not an intended change and could be reverted in the future, when IMPALA-4372
+            is resolved.
+          </p>
+        </li>
+      </ul>
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.6.0" id="incompatible_changes_26x">
+
+    <title>Incompatible Changes Introduced in Impala for CDH 5.8.x / Impala 2.6.x</title>
+
+    <conbody>
+      <ul>
+        <li>
+          <p rev="CDH-41184">
+            The default for the <codeph>RUNTIME_FILTER_MODE</codeph> 
+            query option is changed to <codeph>GLOBAL</codeph> (the highest setting).
+          </p>
+        </li>
+        <li rev="CDH-41184 IMPALA-3007">
+          <p>
+            The <codeph>RUNTIME_BLOOM_FILTER_SIZE</codeph> setting is now only used
+            as a fallback if statistics are not available; otherwise, Impala
+            uses the statistics to estimate the appropriate size to use for each filter.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-3199">
+            Admission control and dynamic resource pools are enabled by default.
+            When upgrading from an earlier release, you must turn on these settings yourself
+            if they are not already enabled.
+            See <xref href="impala_admission.xml#admission_control"/> for details
+            about admission control.
+          </p>
+        </li>
+        <li>
+          <p>
+            Impala reserves some new keywords, in preparation for support for Kudu syntax:
+            <codeph>buckets</codeph>, <codeph>delete</codeph>, <codeph>distribute</codeph>,
+            <codeph>hash</codeph>, <codeph>ignore</codeph>, <codeph>split</codeph>, and <codeph>update</codeph>.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-3554">
+            For Kerberized clusters, the Catalog service now uses
+            the Kerberos principal instead of the operating sytem user that runs
+            the <cmdname>catalogd</cmdname> daemon. 
+            This eliminates the requirement to configure a <codeph>hadoop.user.group.static.mapping.overrides</codeph> 
+            setting to put the OS user into the Sentry administrative group, on clusters where the principal
+            and the OS user name for this user are different.
+          </p>
+        </li>
+        <li>
+          <p>
+            The mechanism for interpreting <codeph>DECIMAL</codeph> literals is
+            improved, no longer going through an intermediate conversion step
+            to <codeph>DOUBLE</codeph>:
+          </p>
+          <ul>
+            <li>
+              <p rev="IMPALA-3163">
+                Casting a <codeph>DECIMAL</codeph> value to <codeph>TIMESTAMP</codeph>
+                <codeph>DOUBLE</codeph> produces a more precise
+                value for the <codeph>TIMESTAMP</codeph> than formerly.
+              </p>
+            </li>
+            <li>
+              <p rev="IMPALA-3439">
+                Certain function calls involving <codeph>DECIMAL</codeph> literals
+                now succeed, when formerly they failed due to lack of a function
+                signature with a <codeph>DOUBLE</codeph> argument.
+              </p>
+            </li>
+          </ul>
+        </li>
+        <li>
+          <p rev="IMPALA-3155">
+            Improved type accuracy for <codeph>CASE</codeph> return values.
+            If all <codeph>WHEN</codeph> clauses of the <codeph>CASE</codeph>
+            expression are of <codeph>CHAR</codeph> type, the final result
+            is also <codeph>CHAR</codeph> instead of being converted to
+            <codeph>STRING</codeph>.
+          </p>
+        </li>
+        <li>
+          <p conref="../shared/impala_common.xml#common/IMPALA-3662"/>
+        </li>
+        <li rev="IMPALA-3452 CDH-39913">
+          <p>
+            The <codeph>S3_SKIP_INSERT_STAGING</codeph> query option, which is enabled by
+            default, increases the speed of <codeph>INSERT</codeph> operations for S3 tables.
+            The speedup applies to regular <codeph>INSERT</codeph>, but not <codeph>INSERT OVERWRITE</codeph>.
+            The tradeoff is the possibility of inconsistent output files left behind if a
+            node fails during <codeph>INSERT</codeph> execution.
+            See <xref href="impala_s3_skip_insert_staging.xml#s3_skip_insert_staging"/> for details.
+          </p>
+        </li>
+      </ul>
+      <p>
+        Certain features are turned off by default, to avoid regressions or unexpected
+        behavior following an upgrade. Consider turning on these features after suitable testing:
+      </p>
+      <ul>
+        <li>
+          <p rev="IMPALA-2660 CDH-40241">
+            Impala now recognizes the <codeph>auth_to_local</codeph> setting,
+            specified through the HDFS configuration setting
+            <codeph>hadoop.security.auth_to_local</codeph>.
+            This feature is disabled by default; to enable it,
+            specify <codeph>--load_auth_to_local_rules=true</codeph>
+            in the <cmdname>impalad</cmdname> configuration settings.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-2069">
+            A new query option, <codeph>PARQUET_ANNOTATE_STRINGS_UTF8</codeph>,
+            makes Impala include the <codeph>UTF-8</codeph> annotation
+            metadata for <codeph>STRING</codeph>, <codeph>CHAR</codeph>,
+            and <codeph>VARCHAR</codeph> columns in Parquet files created
+            by <codeph>INSERT</codeph> or <codeph>CREATE TABLE AS SELECT</codeph>
+            statements.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-2835 CDH-33330">
+            A new query option,
+            <codeph>PARQUET_FALLBACK_SCHEMA_RESOLUTION</codeph>,
+            lets Impala locate columns within Parquet files based on
+            column name rather than ordinal position.
+            This enhancement improves interoperability with applications
+            that write Parquet files with a different order or subset of
+            columns than are used in the Impala table.
+          </p>
+        </li>
+      </ul>
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.5.x" id="incompatible_changes_25x">
+
+    <title>Incompatible Changes Introduced in Impala for CDH 5.7.x / Impala 2.5.x</title>
+
+    <conbody>
+      <ul>
+        <li rev="IMPALA-3044">
+          <p>
+            The admission control default limit for concurrent queries (the <uicontrol>max requests</uicontrol>
+            setting) is now unlimited instead of 200.
+          </p>
+        </li>
+
+        <li>
+          <p rev="IMPALA-2749">
+            Multiplying a mixture of <codeph>DECIMAL</codeph> and <codeph>FLOAT</codeph> or
+            <codeph>DOUBLE</codeph> values now returns
+            <codeph>DOUBLE</codeph> rather than <codeph>DECIMAL</codeph>. This
+            change avoids some cases where an intermediate value would underflow or overflow
+            and become <codeph>NULL</codeph> unexpectedly. The results of
+            multiplying <codeph>DECIMAL</codeph> and <codeph>FLOAT</codeph> or
+            <codeph>DOUBLE</codeph> might now be slightly less precise than
+            before. Previously, the intermediate types and thus the final result
+            depended on the exact order of the values of different types being
+            multiplied, which made the final result values difficult to
+            reason about.
+          </p>
+        </li>
+        <li rev="IMPALA-2204 CDH-33139">
+          <p>
+            Previously, the <codeph>_</codeph> and <codeph>%</codeph> wildcard
+            characters for the <codeph>LIKE</codeph> operator would not match
+            characters on the second or subsequent lines of multi-line string values. The fix for issue
+            <xref href="https://issues.cloudera.org/browse/IMPALA-2204" scope="external" format="html">IMPALA-2204</xref> causes
+            the wildcard matching to apply to the entire string for values
+            containing embedded <codeph>\n</codeph> characters. This could cause
+            different results than in previous Impala releases for identical
+            queries on identical data.
+          </p>
+        </li>
+        <li rev="IMPALA-1748 CDH-38369">
+          <p>
+            Formerly, all Impala UDFs and UDAs required running the
+            <codeph>CREATE FUNCTION</codeph> statements to
+            re-create them after each <cmdname>catalogd</cmdname> restart.
+            In <keyword keyref="impala25_full"/> and higher, functions written in C++ are persisted across
+            restarts, and the requirement to
+            re-create functions only applies to functions written in Java. Adapt any
+            function-reloading logic that you have added to your Impala environment.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-1651">
+              <codeph>CREATE TABLE LIKE</codeph> no longer inherits HDFS caching settings from the source table.
+          </p>
+        </li>
+        <li>
+          <p rev="IMPALA-2070">
+            The <codeph>SHOW DATABASES</codeph> statement now returns two columns rather than one.
+            The second column includes the associated comment string, if any, for each database.
+            Adjust any application code that examines the list of databases and assumes the
+            result set contains only a single column.
+          </p>
+        </li>
+        <li>
+          <p>
+            The output of the <codeph>SHOW FUNCTIONS</codeph> statement includes
+            two new columns, showing the kind of the function (for example,
+            <codeph>BUILTIN</codeph>) and whether or not the function persists
+            across catalog server restarts. For example, the <codeph>SHOW
+            FUNCTIONS</codeph> output for the
+            <codeph>_impala_builtins</codeph> database starts with:
+          </p>
+<codeblock>
++--------------+-------------------------------------------------+-------------+---------------+
+| return type  | signature                                       | binary type | is persistent |
++--------------+-------------------------------------------------+-------------+---------------+
+| BIGINT       | abs(BIGINT)                                     | BUILTIN     | true          |
+| DECIMAL(*,*) | abs(DECIMAL(*,*))                               | BUILTIN     | true          |
+| DOUBLE       | abs(DOUBLE)                                     | BUILTIN     | true          |
+...
+</codeblock>
+        </li>
+      </ul>
+    </conbody>
+
+  </concept>
+
+  <concept rev="2.4.x" id="incompatible_changes_24x">
+
+    <title>Incompatible Changes Introduced in Impala for CDH 5.6.x / Impala 2.4.x</title>
+
+    <conbody>
+      <p>
+        Other than support for DSSD storage, the Impala feature set for CDH 5.6 is the same as for CDH 5.5.
+        Therefore, there are no incompatible changes for Impala introduced in CDH 5.6.
+      </p>
+    </conbody>
+
+  </concept>
+
+<!-- All 2.3.x subsections go under here -->
+
+<!-- Actually for 2.3 / 5.5, let's get away from doing a separate subhead for each maintenance release,
+     because in the normal course of events there will be nothing to add here until 5.6. If something new
+     needs to get noted, just add a new bullet with wording to indicate which 5.5.x release it applies to. -->
+
+  <concept rev="2.3.x" id="incompatible_changes_23x">
+
+    <title>Incompatible Changes Introduced in Impala for CDH 5.5.x / Impala 2.3.x</title>
+
+    <conbody>
+
+      <note conref="../shared/impala_common.xml#common/impala_llama_obsolete"/>
+
+      <ul>
+        <li rev="IMPALA-2005" audience="Cloudera">
+          <p>
+            If a <codeph>CREATE TABLE AS SELECT</codeph> operation fails while data is being inserted,
+            the table is automatically removed. Previously, the table was left behind with no data.
+          </p>
+        </li>
+        <li rev="IMPALA-2130">
+          <p>
+            If Impala encounters a Parquet file that is invalid because of an incorrect magic number,
+            the query skips the file. This change is caused by the fix for issue <xref href="https://issues.cloudera.org/browse/IMPALA-2130" scope="external" format="html">IMPALA-2130</xref>.
+            Previously, Impala would attempt to read the file despite the possibility that the file was corrupted.
+          </p>
+        </li>
+        <li rev="IMPALA-2233 CDH-33145">
+          <p>
+            Previously, calls to overloaded built-in functions could treat parameters as <codeph>DOUBLE</codeph>
+            or <codeph>FLOAT</codeph> when no overload had a signature that matched the exact argument types.
+            Now Impala prefers the function signature with <codeph>DECIMAL</codeph> parameters in this case.
+            This change avoids a possible loss of precision in function calls such as <codeph>greatest(0, 99999.8888)</codeph>;
+            now both parameters are treated as <codeph>DECIMAL</codeph> rather than <codeph>DOUBLE</codeph>, avoiding
+            any loss of precision in the fractional value.
+            This could cause slightly different results than in previous Impala releases for certain function calls.
+          </p>
+        </li>
+        <li rev="IMPALA-1675">
+          <p>
+            Formerly, adding or subtracting a large interval value to a <codeph>TIMESTAMP</codeph> could produce
+            a nonsensical result. Now when the result goes outside the range of <codeph>TIMESTAMP</codeph> values,
+            Impala returns <codeph>NULL</codeph>.
+          </p>
+        </li>
+        <li rev="IMPALA-2251 IMPALA-2257">
+          <p>
+            Formerly, it was possible to accidentally create a table with identical row and column delimiters.
+            This could happen unintentionally, when specifying one of the delimiters and using the
+            default value for the other. Now an attempt to use identical delimiters still succeeds,
+            but displays a warning message.
+          </p>
+        </li>
+        <li rev="CDH-28071">
+          <p>
+            Formerly, Impala could include snippets of table data in log files by default, for example
+            when reporting conversion errors for data values. Now any such log messages are only produced
+            at higher logging levels that you would enable only during debugging.
+          </p>
+        </li>
+<!-- placeholder -->
+      </ul>
+    </conbody>
+
+  </concept>
+
+<!-- All 2.2.x subsections go under here -->
+
+  <concept rev="2.2.x" id="incompatible_changes_22x">
+
+    <title>Incompatible Changes Introduced in Impala for CDH 5.4.x</title>
+
+    <conbody>
+
+<!--
+      <p>
+        No incompatible changes. CDH maintenance releases such as 5.4.1, 5.4.2, and so on are exclusively bug fix releases.
+        See <xref href="impala_incompatible_changes.xml#incompatible_changes_220"/> for the initial Impala feature release,
+        which are the releases that typically include incompatible changes.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_22x"/>
+
+    </conbody>
+  </concept>
+
+  <concept rev="2.2.0" id="incompatible_changes_220">
+
+    <title>Incompatible Changes Introduced in Impala 2.2.0 / CDH 5.4.0</title>
+
+    <conbody>
+-->
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_220"/>
+
+      <section id="files_220">
+      <title>
+        Changes to File Handling
+      </title>
+        <p conref="../shared/impala_common.xml#common/ignore_file_extensions"/>
+        <p>
+          The log rotation feature in Impala 2.2.0 and higher
+          means that older log files are now removed by default.
+          The default is to preserve the latest 10 log files for each
+          severity level, for each Impala-related daemon. If you have
+          set up your own log rotation processes that expect older
+          files to be present, either adjust your procedures or
+          change the Impala <codeph>-max_log_files</codeph> setting.
+          <ph audience="PDF">See <xref href="impala_logging.xml#logs_rotate"/> for details.</ph>
+        </p>
+      </section>
+
+
+      <section id="prereqs_210">
+      <title>
+        Changes to Prerequisites
+      </title>
+        <p conref="../shared/impala_common.xml#common/cpu_prereq"/>
+      </section>
+
+    </conbody>
+  </concept>
+
+<!-- All 2.1.x subsections go under here -->
+
+  <concept rev="2.1.x" id="incompatible_changes_21x">
+
+    <title>Incompatible Changes Introduced in Impala for CDH 5.3.x</title>
+
+    <conbody>
+
+<!--
+      <p>
+        No incompatible changes. CDH maintenance releases such as 5.3.1, 5.3.2, and so on are exclusively bug fix releases.
+        See <xref href="impala_incompatible_changes.xml#incompatible_changes_210"/> for the initial Impala feature release,
+        which are the releases that typically include incompatible changes.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+    </conbody>
+  </concept>
+
+  <concept rev="2.1.7" id="incompatible_changes_217">
+    <title>Incompatible Changes Introduced in Cloudera Impala 2.1.7 / CDH 5.3.9</title>
+    <conbody>
+      <p>
+        No incompatible changes.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+    </conbody>
+  </concept>
+
+  <concept rev="2.1.6" id="incompatible_changes_216">
+    <title>Incompatible Changes Introduced in Impala 2.1.6 / CDH 5.3.8</title>
+    <conbody>
+      <p>
+        No incompatible changes.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+    </conbody>
+  </concept>
+
+  <concept rev="2.1.5" id="incompatible_changes_215">
+    <title>Incompatible Changes Introduced in Impala 2.1.5 / CDH 5.3.6</title>
+    <conbody>
+      <p>
+        No incompatible changes.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+    </conbody>
+  </concept>
+
+  <concept rev="2.1.4" id="incompatible_changes_214">
+    <title>Incompatible Changes Introduced in Impala 2.1.4 / CDH 5.3.4</title>
+    <conbody>
+      <p>
+        No incompatible changes.
+        <ph conref="../shared/impala_common.xml#common/impala_214_redux"/>
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/>
+
+    </conbody>
+  </concept>
+
+  <concept rev="2.1.3" id="incompatible_changes_213">
+    <title>Incompatible Changes Introduced in Impala 2.1.3 / CDH 5.3.3</title>
+    <conbody>
+      <p>
+        No incompatible changes.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_213"/>
+
+    </conbody>
+  </concept>
+
+  <concept rev="2.1.2" id="incompatible_changes_212">
+    <title>Incompatible Changes Introduced in Impala 2.1.2 / CDH 5.3.2</title>
+    <conbody>
+      <p>
+        No incompatible changes.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_212"/>
+
+    </conbody>
+  </concept>
+
+  <concept rev="2.1.1" id="incompatible_changes_211">
+
+    <title>Incompatible Changes Introduced in Impala 2.1.1 / CDH 5.3.1</title>
+
+    <conbody>
+
+      <p>
+        No incompatible changes.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept rev="2.1.0" id="incompatible_changes_210">
+
+    <title>Incompatible Changes Introduced in Impala 2.1.0 / CDH 5.3.0</title>
+
+    <conbody>
+-->
+
+      <section id="prereqs_210">
+      <title>
+        Changes to Prerequisites
+      </title>
+        <p rev="CDH-24874">
+          Currently, Impala 2.1.x does not function on CPUs without the SSE4.1 instruction set. This minimum CPU
+          requirement is higher than in previous versions, which relied on the older SSSE3 instruction set. Check
+          the CPU level of the hosts in your cluster before upgrading to Impala 2.1.x or CDH 5.3.x.
+        </p>
+      </section>
+
+      <section id="output_format_210">
+      <title>
+        Changes to Output Format
+      </title>
+        <p>
+          The <q>small query</q> optimization feature introduces some new information in the
+          <codeph>EXPLAIN</codeph> plan, which you might need to account for if you parse the text of the plan
+          output.
+        </p>
+      </section>
+
+      <section id="reserved_words_210">
+      <title>
+        New Reserved Words
+      </title>
+      <p>
+        New SQL syntax introduces additional reserved words:
+        <codeph>FOR</codeph>, <codeph>GRANT</codeph>, <codeph>REVOKE</codeph>, <codeph>ROLE</codeph>, <codeph>ROLES</codeph>,
+        <codeph>INCREMENTAL</codeph>.
+        <ph audience="PDF">As always, see <xref href="impala_reserved_words.xml#reserved_words"/>
+        for the set of reserved words for the current release, and the quoting techniques to avoid name conflicts.</ph>
+      </p>
+      </section>
+    </conbody>
+  </concept>
+
+<!-- All 2.0.x subsections go under here -->
+
+  <concept rev="2.0.5" id="incompatible_changes_205">
+
+    <title>Incompatible Changes Introduced in Impala 2.0.5 / CDH 5.2.6</title>
+
+    <conbody>
+
+      <p>
+        No incompatible changes.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_205"/>
+
+    </conbody>
+  </concept>
+
+  <concept rev="2.0.4" id="incompatible_changes_204">
+
+    <title>Incompatible Changes Introduced in Impala 2.0.4 / CDH 5.2.5</title>
+
+    <conbody>
+
+      <p>
+        No incompatible changes.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_204"/>
+
+    </conbody>
+  </concept>
+
+  <concept rev="2.0.3" id="incompatible_changes_203">
+
+    <title>Incompatible Changes Introduced in Impala 2.0.3 / CDH 5.2.4</title>
+
+    <conbody>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_203"/>
+
+    </conbody>
+  </concept>
+
+  <concept rev="2.0.2" id="incompatible_changes_202">
+
+    <title>Incompatible Changes Introduced in Impala 2.0.2 / CDH 5.2.3</title>
+
+    <conbody>
+
+      <p>
+        No incompatible changes.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_202"/>
+
+    </conbody>
+  </concept>
+
+  <concept rev="2.0.1" id="incompatible_changes_201">
+
+    <title>Incompatible Changes Introduced in Impala 2.0.1 / CDH 5.2.1</title>
+
+    <conbody>
+
+      <ul>
+        <li>
+          <p conref="../shared/impala_common.xml#common/insert_hidden_work_directory"/>
+        </li>
+
+        <li>
+          <p>
+            The <codeph>abs()</codeph> function now takes a broader range of numeric types as arguments, and the
+            return type is the same as the argument type.
+          </p>
+        </li>
+
+        <li>
+          <p>
+            Shorthand notation for character classes in regular expressions, such as <codeph>\d</codeph> for digit,
+            are now available again in regular expression operators and functions such as
+            <codeph>regexp_extract()</codeph> and <codeph>regexp_replace()</codeph>. Some other differences in
+            regular expression behavior remain between Impala 1.x and Impala 2.x releases. See
+            <xref href="impala_incompatible_changes.xml#incompatible_changes_200"/> for details.
+          </p>
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept rev="2.0.0" id="incompatible_changes_200">
+
+    <title>Incompatible Changes Introduced in Impala 2.0.0 / CDH 5.2.0</title>
+
+    <conbody>
+
+      <section id="prereqs_200">
+      <title>
+        Changes to Prerequisites
+      </title>
+        <p rev="CDH-24874">
+          Currently, Impala 2.0.x does not function on CPUs without the SSE4.1 instruction set. This minimum CPU
+          requirement is higher than in previous versions, which relied on the older SSSE3 instruction set. Check
+          the CPU level of the hosts in your cluster before upgrading to Impala 2.0.x or CDH 5.2.x.
+        </p>
+      </section>
+
+      <section id="queries_200">
+      <title>
+        Changes to Query Syntax
+      </title>
+
+        <p>
+          The new syntax where query hints are allowed in comments causes some changes in the way comments are
+          parsed in the <cmdname>impala-shell</cmdname> interpreter. Previously, you could end a
+          <codeph>--</codeph> comment line with a semicolon and <cmdname>impala-shell</cmdname> would treat that
+          as a no-op statement. Now, a comment line ending with a semicolon is passed as an empty statement to
+          the Impala daemon, where it is flagged as an error.
+        </p>
+
+        <p>
+          Impala 2.0 and later uses a different support library for regular expression parsing than in earlier
+          Impala versions. Now, Impala uses the
+          <xref href="https://code.google.com/p/re2/" scope="external" format="html">Google RE2 library</xref>
+          rather than Boost for evaluating regular expressions. This implementation change causes some
+          differences in the allowed regular expression syntax, and in the way certain regex operators are
+          interpreted. The following are some of the major differences (not necessarily a complete list):
+        </p>
+        <ul>
+          <li>
+            <p>
+              <codeph>.*?</codeph> notation for non-greedy matches is now supported, where it was not in earlier
+              Impala releases.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              By default, <codeph>^</codeph> and <codeph>$</codeph> now match only begin/end of buffer, not
+              begin/end of each line. This behavior can be overridden in the regex itself using the
+              <codeph>m</codeph> flag.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              By default, <codeph>.</codeph> does not match newline. This behavior can be overridden in the regex
+              itself using the <codeph>s</codeph> flag.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              <codeph>\Z</codeph> is not supported.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              <codeph>&lt;</codeph> and <codeph>&gt;</codeph> for start of word and end of word are not
+              supported.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              Lookahead and lookbehind are not supported.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              Shorthand notation for character classes, such as <codeph>\d</codeph> for digit, is not recognized.
+              (This restriction is lifted in Impala 2.0.1, which restores the shorthand notation.)
+            </p>
+          </li>
+        </ul>
+      </section>
+
+      <section id="output_format_210">
+      <title>
+        Changes to Output Format
+      </title>
+
+        <p conref="../shared/impala_common.xml#common/user_kerberized"/>
+
+        <p>
+          The changed format for the user name in secure environments is also reflected where the user name is
+          displayed in the output of the <codeph>PROFILE</codeph> command.
+        </p>
+
+        <p>
+          In the output from <codeph>SHOW FUNCTIONS</codeph>, <codeph>SHOW AGGREGATE FUNCTIONS</codeph>, and
+          <codeph>SHOW ANALYTIC FUNCTIONS</codeph>, arguments and return types of arbitrary
+          <codeph>DECIMAL</codeph> scale and precision are represented as <codeph>DECIMAL(*,*)</codeph>.
+          Formerly, these items were displayed as <codeph>DECIMAL(-1,-1)</codeph>.
+        </p>
+
+      </section>
+
+      <section id="query_options_200">
+      <title>
+        Changes to Query Options
+      </title>
+        <p>
+          The <codeph>PARQUET_COMPRESSION_CODEC</codeph> query option has been replaced by the
+          <codeph>COMPRESSION_CODEC</codeph> query option.
+          <ph audience="PDF">See <xref href="impala_compression_codec.xml#compression_codec"/> for details.</ph>
+        </p>
+      </section>
+
+      <section id="config_options_200">
+      <title>
+        Changes to Configuration Options
+      </title>
+
+        <p>
+          The meaning of the <codeph>--idle_query_timeout</codeph> configuration option is changed, to
+          accommodate the new <codeph>QUERY_TIMEOUT_S</codeph> query option. Rather than setting an absolute
+          timeout period that applies to all queries, it now sets a maximum timeout period, which can be adjusted
+          downward for individual queries by specifying a value for the <codeph>QUERY_TIMEOUT_S</codeph> query
+          option. In sessions where no <codeph>QUERY_TIMEOUT_S</codeph> query option is specified, the
+          <codeph>--idle_query_timeout</codeph> timeout period applies the same as in earlier versions.
+        </p>
+
+        <p>
+          The <codeph>--strict_unicode</codeph> option of <cmdname>impala-shell</cmdname> was removed. To avoid
+          problems with Unicode values in <cmdname>impala-shell</cmdname>, define the following locale setting
+          before running <cmdname>impala-shell</cmdname>:
+        </p>
+<codeblock>export LC_CTYPE=en_US.UTF-8
+</codeblock>
+
+      </section>
+
+      <section id="reserved_words_210">
+      <title>
+        New Reserved Words
+      </title>
+        <p>
+          Some new SQL syntax requires the addition of new reserved words: <codeph>ANTI</codeph>,
+          <codeph>ANALYTIC</codeph>, <codeph>OVER</codeph>, <codeph>PRECEDING</codeph>,
+          <codeph>UNBOUNDED</codeph>, <codeph>FOLLOWING</codeph>, <codeph>CURRENT</codeph>,
+          <codeph>ROWS</codeph>, <codeph>RANGE</codeph>, <codeph>CHAR</codeph>, <codeph>VARCHAR</codeph>.
+          <ph audience="PDF">As always, see <xref href="impala_reserved_words.xml#reserved_words"/>
+          for the set of reserved words for the current release, and the quoting techniques to avoid name conflicts.</ph>
+        </p>
+      </section>
+
+      <section id="output_files_200">
+      <title>
+        Changes to Data Files
+      </title>
+
+        <p id="parquet_block_size">
+          The default Parquet block size for Impala is changed from 1 GB to 256 MB. This change could have
+          implications for the sizes of Parquet files produced by <codeph>INSERT</codeph> and <codeph>CREATE
+          TABLE AS SELECT</codeph> statements.
+        </p>
+        <p>
+          Although older Impala releases typically produced files that were smaller than the old default size of
+          1 GB, now the file size matches more closely whatever value is specified for the
+          <codeph>PARQUET_FILE_SIZE</codeph> query option. Thus, if you use a non-default value for this setting,
+          the output files could be larger than before. They still might be somewhat smaller than the specified
+          value, because Impala makes conservative estimates about the space needed to represent each column as
+          it encodes the data.
+        </p>
+        <p>
+          When you do not specify an explicit value for the <codeph>PARQUET_FILE_SIZE</codeph> query option,
+          Impala tries to keep the file size within the 256 MB default size, but Impala might adjust the file
+          size to be somewhat larger if needed to accommodate the layout for <term>wide</term> tables, that is,
+          tables with hundreds or thousands of columns.
+        </p>
+        <p>
+          This change is unlikely to affect memory usage while writing Parquet files, because Impala does not
+          pre-allocate the memory needed to hold the entire Parquet block.
+        </p>
+
+      </section>
+
+    </conbody>
+  </concept>
+
+  <concept rev="1.4.4" id="incompatible_changes_144">
+    <title>Incompatible Changes Introduced in Impala 1.4.4 / CDH 5.1.5</title>
+    <conbody>
+      <p>
+        No incompatible changes.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_144"/>
+
+    </conbody>
+  </concept>
+
+  <concept rev="1.4.3" id="incompatible_changes_143">
+
+    <title>Incompatible Changes Introduced in Impala 1.4.3 / CDH 5.1.4</title>
+
+    <conbody>
+
+      <p>
+        No incompatible changes. The TLS/SSL security fix does not require any change in the way you interact with
+        Impala.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_143"/>
+
+    </conbody>
+  </concept>
+
+  <concept rev="1.4.2" id="incompatible_changes_142">
+
+    <title>Incompatible Changes Introduced in Impala 1.4.2 / CDH 5.1.3</title>
+
+    <conbody>
+
+      <p>
+        None. Impala 1.4.2 is purely a bug-fix release. It does not include any incompatible changes.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_142"/>
+
+    </conbody>
+  </concept>
+
+  <concept rev="1.4.1" id="incompatible_changes_141">
+
+    <title>Incompatible Changes Introduced in Impala 1.4.1 / CDH 5.1.2</title>
+
+    <conbody>
+
+      <p>
+        None. Impala 1.4.1 is purely a bug-fix release. It does not include any incompatible changes.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept rev="1.4.0" id="incompatible_changes_140">
+
+    <title>Incompatible Changes Introduced in Impala 1.4.0 / CDH 5.1.0</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Deprecated Features"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <ul>
+        <li>
+          <p>
+            There is a slight change to required security privileges in the Sentry framework. To create a new
+            object, now you need the <codeph>ALL</codeph> privilege on the parent object. For example, to create a
+            new table, view, or function requires having the <codeph>ALL</codeph> privilege on the database
+            containing the new object. <ph audience="PDF">See
+              <xref
+                href="http://www.cloudera.com/documentation/enterprise/latest/topics/impala_authorization.html"
+                scope="external" format="html">Privilege Model and Object Hierarchy</xref>
+              for a full list of operations and
+            associated privileges.</ph>
+          </p>
+        </li>
+
+        <li>
+          <p>
+            With the ability of <codeph>ORDER BY</codeph> queries to process unlimited amounts of data with no
+            <codeph>LIMIT</codeph> clause, the query options <codeph>DEFAULT_ORDER_BY_LIMIT</codeph> and
+            <codeph>ABORT_ON_DEFAULT_LIMIT_EXCEEDED</codeph> are now deprecated and have no effect.
+            <ph audience="PDF">See <xref href="impala_order_by.xml#order_by"/> for details about improvements to
+            the <codeph>ORDER BY</codeph> clause.</ph>
+          </p>
+        </li>
+
+        <li>
+          <p>
+            There are some changes to the list of reserved words. <ph audience="PDF">See
+            <xref href="impala_reserved_words.xml#reserved_words"/> for the most current list.</ph> The following
+            keywords are new:
+          </p>
+          <ul>
+            <li>
+              <codeph>API_VERSION</codeph>
+            </li>
+
+            <li>
+              <codeph>BINARY</codeph>
+            </li>
+
+            <li>
+              <codeph>CACHED</codeph>
+            </li>
+
+            <li>
+              <codeph>CLASS</codeph>
+            </li>
+
+            <li>
+              <codeph>PARTITIONS</codeph>
+            </li>
+
+            <li>
+              <codeph>PRODUCED</codeph>
+            </li>
+
+            <li>
+              <codeph>UNCACHED</codeph>
+            </li>
+          </ul>
+          <p>
+            The following were formerly reserved keywords, but are no longer reserved:
+          </p>
+          <ul>
+            <li>
+              <codeph>COUNT</codeph>
+            </li>
+
+            <li>
+              <codeph>GROUP_CONCAT</codeph>
+            </li>
+
+            <li>
+              <codeph>NDV</codeph>
+            </li>
+
+            <li>
+              <codeph>SUM</codeph>
+            </li>
+          </ul>
+        </li>
+
+        <li>
+          <p>
+            The fix for issue
+            <xref href="https://issues.cloudera.org/browse/IMPALA-973" scope="external" format="html">IMPALA-973</xref>
+            changes the behavior of the <codeph>INVALIDATE METADATA</codeph> statement regarding nonexistent
+            tables. In Impala 1.4.0 and higher, the statement returns an error if the specified table is not in the
+            metastore database at all. It completes successfully if the specified table is in the metastore
+            database but not yet recognized by Impala, for example if the table was created through Hive. Formerly,
+            you could issue this statement for a completely nonexistent table, with no error.
+          </p>
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept rev="1.3.3" id="incompatible_changes_133">
+
+    <title>Incompatible Changes Introduced in Impala 1.3.3 / CDH 5.0.5</title>
+
+    <conbody>
+
+      <p>
+        No incompatible changes. The TLS/SSL security fix does not require any change in the way you interact with
+        Impala.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_133"/>
+
+    </conbody>
+  </concept>
+
+  <concept rev="1.3.2" id="incompatible_changes_132">
+
+    <title>Incompatible Changes Introduced in Impala 1.3.2 / CDH 5.0.4</title>
+
+    <conbody>
+
+      <p>
+        With the fix for IMPALA-1019, you can use HDFS caching for files that are accessed by Impala.
+      </p>
+
+      <note conref="../shared/impala_common.xml#common/only_cdh5_132"/>
+
+    </conbody>
+  </concept>
+
+  <concept rev="1.3.1" id="incompatible_changes_131">
+
+    <title>Incompatible Changes Introduced in Impala 1.3.1 / CDH 5.0.3</title>
+
+    <conbody>
+
+      <ul>
+        <li>
+          <p conref="../shared/impala_common.xml#common/regexp_matching"/>
+        </li>
+
+        <li>
+          <p>
+            The result set for the <codeph>SHOW FUNCTIONS</codeph> statement includes a new first column, with the
+            data type of the return value. <ph audience="PDF">See <xref href="impala_show.xml#show"/> for
+            examples.</ph>
+          </p>
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept rev="1.3.0" id="incompatible_changes_130">
+
+    <title>Incompatible Changes Introduced in Impala 1.3.0 / CDH 5.0.0</title>
+
+    <conbody>
+
+      <ul>
+        <li>
+          <p>
+            The <codeph>EXPLAIN_LEVEL</codeph> query option now accepts numeric options from 0 (most concise) to 3
+            (most verbose), rather than only 0 or 1. If you formerly used <codeph>SET EXPLAIN_LEVEL=1</codeph> to
+            get detailed explain plans, switch to <codeph>SET EXPLAIN_LEVEL=3</codeph>. If you used the mnemonic
+            keyword (<codeph>SET EXPLAIN_LEVEL=verbose</codeph>), you do not need to change your code because now
+            level 3 corresponds to <codeph>verbose</codeph>. <ph audience="PDF">See
+            <xref href="impala_explain_level.xml#explain_level"/> for details about the allowed explain levels, and
+            <xref href="impala_explain_plan.xml#explain_plan"/> for usage information.</ph>
+          </p>
+        </li>
+
+        <li>
+          <p>
+            The keyword <codeph>DECIMAL</codeph> is now a reserved word. If you have any databases, tables,
+            columns, or other objects already named <codeph>DECIMAL</codeph>, quote any references to them using
+            backticks (<codeph>``</codeph>) to avoid name conflicts with the keyword.
+            <note>
+              Although the <codeph>DECIMAL</codeph> keyword is a reserved word, currently Impala does not support
+              <codeph>DECIMAL</codeph> as a data type for columns.
+            </note>
+          </p>
+        </li>
+
+        <li>
+          <p>
+            The query option named <codeph>YARN_POOL</codeph> during the CDH 5 beta period is now named
+            <codeph>REQUEST_POOL</codeph> to reflect its broader use with the Impala admission control feature.
+            <ph audience="PDF">See <xref href="impala_request_pool.xml#request_pool"/> for information about the
+            option, and <xref href="impala_admission.xml#admission_control"/> for details about its use with the
+            admission control feature.</ph>
+          </p>
+        </li>
+
+        <li>
+          <p>
+            There are some changes to the list of reserved words. <ph audience="PDF">See
+            <xref href="impala_reserved_words.xml#reserved_words"/> for the most current list.</ph>
+          </p>
+          <ul>
+            <li>
+              <p>
+                The names of aggregate functions are no longer reserved words, so you can have databases, tables,
+                columns, or other objects named <codeph>AVG</codeph>, <codeph>MIN</codeph>, and so on without any
+                name conflicts.
+              </p>
+            </li>
+
+            <li>
+              <p>
+                The internal function names <codeph>DISTINCTPC</codeph> and <codeph>DISTINCTPCSA</codeph> are no
+                longer reserved words, although <codeph>DISTINCT</codeph> is still a reserved word.
+              </p>
+            </li>
+
+            <li>
+              <p>
+                The keywords <codeph>CLOSE_FN</codeph> and <codeph>PREPARE_FN</codeph> are now reserved words.
+                <ph audience="PDF">See <xref href="impala_create_function.xml#create_function"/> for their role in
+                the <codeph>CREATE FUNCTION</codeph> statement, and <xref href="impala_udf.xml#udf_threads"/> for
+                usage information.</ph>
+              </p>
+            </li>
+          </ul>
+        </li>
+
+        <li>
+          <p>
+            The HDFS property <codeph>dfs.client.file-block-storage-locations.timeout</codeph> was renamed to
+            <codeph>dfs.client.file-block-storage-locations.timeout.millis</codeph>, to emphasize that the unit of
+            measure is milliseconds, not seconds. Impala requires a timeout of at least 10 seconds, making the
+            minimum value for this setting 10000. On systems not managed by Cloudera Manager, you might need to
+            edit the <filepath>hdfs-site.xml</filepath> file in the Impala configuration directory for the new name
+            and minimum value.
+          </p>
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept rev="1.2.4" id="incompatible_changes_124">
+
+    <title>Incompatible Changes Introduced in Impala 1.2.4</title>
+
+    <conbody>
+
+      <p>
+        There are no incompatible changes introduced in Impala 1.2.4.
+      </p>
+
+      <p>
+        Previously, after creating a table in Hive, you had to issue the <codeph>INVALIDATE METADATA</codeph>
+        statement with no table name, a potentially expensive operation on clusters with many databases, tables,
+        and partitions. Starting in Impala 1.2.4, you can issue the statement <codeph>INVALIDATE METADATA
+        <varname>table_name</varname></codeph> for a table newly created through Hive. Loading the metadata for
+        only this one table is faster and involves less network overhead. Therefore, you might revisit your setup
+        DDL scripts to add the table name to <codeph>INVALIDATE METADATA</codeph> statements, in cases where you
+        create and populate the tables through Hive before querying them through Impala.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept rev="1.2.3" id="incompatible_changes_123">
+
+    <title>Incompatible Changes Introduced in Impala 1.2.3</title>
+
+    <conbody>
+
+      <p>
+        Because the feature set of Impala 1.2.3 is identical to Impala 1.2.2, there are no new incompatible
+        changes. See <xref href="impala_incompatible_changes.xml#incompatible_changes_122"/> if you are upgrading
+        from Impala 1.2.1 or 1.1.x.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept rev="1.2.2" id="incompatible_changes_122">
+
+    <title>Incompatible Changes Introduced in Impala 1.2.2</title>
+
+    <conbody>
+
+      <p>
+        The following changes to SQL syntax and semantics in Impala 1.2.2 could require updates to your SQL code,
+        or schema objects such as tables or views:
+      </p>
+
+      <ul>
+        <li>
+          <p>
+            With the addition of the <codeph>CROSS JOIN</codeph> keyword, you might need to rewrite any queries
+            that refer to a table named <codeph>CROSS</codeph> or use the name <codeph>CROSS</codeph> as a table
+            alias:
+          </p>
+<codeblock>-- Formerly, 'cross' in this query was an alias for t1
+-- and it was a normal join query.
+-- In 1.2.2 and higher, CROSS JOIN is a keyword, so 'cross'
+-- is not interpreted as a table alias, and the query
+-- uses the special CROSS JOIN processing rather than a
+-- regular join.
+select * from t1 cross join t2...
+
+-- Now if CROSS is used in other context such as a table or column name,
+-- use backticks to escape it.
+create table `cross` (x int);
+select * from `cross`;</codeblock>
+        </li>
+
+        <li>
+          <p>
+            Formerly, a <codeph>DROP DATABASE</codeph> statement in Impala would not remove the top-level HDFS
+            directory for that database. The <codeph>DROP DATABASE</codeph> has been enhanced to remove that
+            directory. (You still need to drop all the tables inside the database first; this change only applies
+            to the top-level directory for the entire database.)
+          </p>
+        </li>
+
+        <li>
+          The keyword <codeph>PARQUET</codeph> is introduced as a synonym for <codeph>PARQUETFILE</codeph> in the
+          <codeph>CREATE TABLE</codeph> and <codeph>ALTER TABLE</codeph> statements, because that is the common
+          name for the file format. (As opposed to SequenceFile and RCFile where the <q>File</q> suffix is part of
+          the name.) Documentation examples have been changed to prefer the new shorter keyword. The
+          <codeph>PARQUETFILE</codeph> keyword is still available for backward compatibility with older Impala
+          versions.
+        </li>
+
+        <li>
+          New overloads are available for several operators and built-in functions, allowing you to insert their
+          result values into smaller numeric columns such as <codeph>INT</codeph>, <codeph>SMALLINT</codeph>,
+          <codeph>TINYINT</codeph>, and <codeph>FLOAT</codeph> without using a <codeph>CAST()</codeph> call. If you
+          remove the <codeph>CAST()</codeph> calls from <codeph>INSERT</codeph> statements, those statements might
+          not work with earlier versions of Impala.
+        </li>
+      </ul>
+
+      <p>
+        Because many users are likely to upgrade straight from Impala 1.x to Impala 1.2.2, also read
+        <xref href="impala_incompatible_changes.xml#incompatible_changes_121"/> for things to note about upgrading
+        to Impala 1.2.x in general.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/cm48_upgrade"/>
+
+<!-- <note conref="common.xml#common/cdh4_cdh5_upgrade"/> -->
+    </conbody>
+  </concept>
+
+  <concept rev="1.2.1" id="incompatible_changes_121">
+
+    <title>Incompatible Changes Introduced in Impala 1.2.1</title>
+
+    <conbody>
+
+      <p>
+        The following changes to SQL syntax and semantics in Impala 1.2.1 could require updates to your SQL code,
+        or schema objects such as tables or views:
+      </p>
+
+      <ul>
+        <li>
+          <p conref="../shared/impala_common.xml#common/null_sorting_change"/>
+          <p audience="PDF">
+            See <xref href="impala_literals.xml#null"/> for more information.
+          </p>
+        </li>
+      </ul>
+
+      <p>
+        Impala 1.2.1 goes along with CDH 4.5 and Cloudera Manager 4.8. If you used the beta version Impala 1.2.0
+        that came with the beta of CDH 5, Impala 1.2.1 includes all the features of Impala 1.2.0 except for
+        resource management, which relies on the YARN framework from CDH 5.
+      </p>
+
+      <p>
+        The new <cmdname>catalogd</cmdname> service might require changes to any user-written scripts that stop,
+        start, or restart Impala services, install or upgrade Impala packages, or issue <codeph>REFRESH</codeph> or
+        <codeph>INVALIDATE METADATA</codeph> statements:
+      </p>
+
+      <ul conref="../shared/impala_common.xml#common/catalogd_xrefs">
+        <li/>
+      </ul>
+
+      <p conref="../shared/impala_common.xml#common/cm48_upgrade"/>
+
+<!--  <note conref="common.xml#common/cdh4_cdh5_upgrade"/> -->
+    </conbody>
+  </concept>
+
+  <concept rev="1.2" id="incompatible_changes_120">
+
+    <title>Incompatible Changes Introduced in Impala 1.2.0 (Beta)</title>
+
+    <conbody>
+
+      <p>
+        There are no incompatible changes to SQL syntax in Impala 1.2.0 (beta).
+      </p>
+
+      <p>
+        Because Impala 1.2.0 is bundled with the CDH 5 beta download and depends on specific levels of Apache
+        Hadoop components supplied with CDH 5, you can only install it in combination with the CDH 5 beta.
+      </p>
+
+      <p>
+        The new <cmdname>catalogd</cmdname> service might require changes to any user-written scripts that stop,
+        start, or restart Impala services, install or upgrade Impala packages, or issue <codeph>REFRESH</codeph> or
+        <codeph>INVALIDATE METADATA</codeph> statements:
+      </p>
+
+      <ul conref="../shared/impala_common.xml#common/catalogd_xrefs">
+        <li/>
+      </ul>
+
+      <p>
+        The new resource management feature interacts with both YARN and Llama services, which are available in CDH
+        5. These services are set up for you automatically in a Cloudera Manager (CM) environment. For information
+        about setting up the YARN and Llama services, see the instructions for
+<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh_ig_yarn_cluster_deploy.html -->
+        <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_yarn_cluster_deploy.html" scope="external" format="html">YARN</xref>
+        and
+<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh_ig_llama_installation.html -->
+<!-- Then this, which was removed: http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_llama_installation.html -->
+        <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/admin_llama.html" scope="external" format="html">Llama</xref>
+        in the <cite>CDH 5 Documentation</cite>. <ph audience="PDF">See
+        <xref href="impala_resource_management.xml#resource_management"/> for usage information for Impala resource
+        management.</ph>
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="incompatible_changes_111">
+
+    <title>Incompatible Changes Introduced in Impala 1.1.1</title>
+
+    <conbody>
+
+      <p>
+        There are no incompatible changes in Impala 1.1.1.
+      </p>
+
+<!-- These couple of paragraphs were originally intended to be conref'ed from the Parquet section of Installing/Using. -->
+
+<!-- But conbodydiv tag too restrictive, can't have just paragraphs and codeblocks inside. -->
+
+<!-- So I will physically copy the info for the time being. -->
+
+<!-- Also copying it under the Upgrading topic. -->
+
+<!-- <conbodydiv conref="impala_parquet.xml#upgrade_parquet_metadata"/> -->
+
+      <p>
+        Previously, it was not possible to create Parquet data through Impala and reuse that table within Hive. Now
+        that Parquet support is available for Hive 10, reusing existing Impala Parquet data files in Hive requires
+        updating the table metadata. Use the following command if you are already running Impala 1.1.1:
+      </p>
+
+<codeblock>ALTER TABLE <varname>table_name</varname> SET FILEFORMAT PARQUETFILE;
+</codeblock>
+
+      <p>
+        If you are running a level of Impala that is older than 1.1.1, do the metadata update through Hive:
+      </p>
+
+<codeblock>ALTER TABLE <varname>table_name</varname> SET SERDE 'parquet.hive.serde.ParquetHiveSerDe';
+ALTER TABLE <varname>table_name</varname> SET FILEFORMAT
+  INPUTFORMAT "parquet.hive.DeprecatedParquetInputFormat"
+  OUTPUTFORMAT "parquet.hive.DeprecatedParquetOutputFormat";
+</codeblock>
+
+      <p>
+        Impala 1.1.1 and higher can reuse Parquet data files created by Hive, without any action required.
+      </p>
+
+      <p>
+        As usual, make sure to upgrade the <codeph>impala-lzo-cdh4</codeph> package to the latest level at the same
+        time as you upgrade the Impala server.
+      </p>
+    </conbody>
+  </concept>
+
+  <concept id="incompatible_changes_11">
+
+    <title>Incompatible Change Introduced in Impala 1.1</title>
+
+    <conbody>
+
+      <ul>
+        <li>
+          <p>
+            The <codeph>REFRESH</codeph> statement now requires a table name; in Impala 1.0, the table name was
+            optional. This syntax change is part of the internal rework to make <codeph>REFRESH</codeph> a true
+            Impala SQL statement so that it can be called through the JDBC and ODBC APIs. <codeph>REFRESH</codeph>
+            now reloads the metadata immediately, rather than marking it for update the next time any affected
+            table is accessed. The previous behavior, where omitting the table name caused a refresh of the entire
+            Impala metadata catalog, is available through the new <codeph>INVALIDATE METADATA</codeph> statement.
+            <codeph>INVALIDATE METADATA</codeph> can be specified with a table name to affect a single table, or
+            without a table name to affect the entire metadata catalog; the relevant metadata is reloaded the next
+            time it is requested during the processing for a SQL statement. See
+            <xref href="impala_refresh.xml#refresh"/> and
+            <xref href="impala_invalidate_metadata.xml#invalidate_metadata"/> for the latest details about these
+            statements.
+          </p>
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="incompatible_changes_10">
+
+    <title>Incompatible Changes Introduced in Impala 1.0</title>
+
+    <conbody>
+
+      <ul>
+        <li>
+          If you use LZO-compressed text files, when you upgrade Impala to version 1.0, also update the
+          <codeph>impala-lzo-cdh4</codeph> to the latest level. See <xref href="impala_txtfile.xml#lzo"/> for
+          details.
+        </li>
+
+        <li>
+          Cloudera Manager 4.5.2 and higher only supports Impala 1.0 and higher, and vice versa. If you upgrade to
+          Impala 1.0 or higher managed by Cloudera Manager, you must also upgrade Cloudera Manager to version 4.5.2
+          or higher. If you upgrade from an earlier version of Cloudera Manager, and were using Impala, you must
+          also upgrade Impala to version 1.0 or higher. The beta versions of Impala are no longer supported as of
+          the release of Impala 1.0.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="incompatible_changes_07">
+
+    <title>Incompatible Change Introduced in Version 0.7 of the Impala Beta Release</title>
+
+    <conbody>
+
+      <ul>
+        <li>
+          The defaults for the <codeph>-nn</codeph> and <codeph>-nn_port</codeph> flags have changed and are now
+          read from <codeph>core-site.xml</codeph>. Impala prints the values of <codeph>-nn</codeph> and
+          <codeph>-nn_port</codeph> to the log when it starts. The ability to set <codeph>-nn</codeph> and
+          <codeph>-nn_port</codeph> on the command line is deprecated in 0.7 and may be removed in Impala 0.8.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="incompatible_changes_06">
+
+    <title>Incompatible Change Introduced in Version 0.6 of the Impala Beta Release</title>
+
+    <conbody>
+
+      <ul>
+        <li>
+          Cloudera Manager 4.5 supports only version 0.6 of the Impala Beta Release. It does not support
+          the earlier beta versions. If you upgrade your Cloudera Manager installation, you must also upgrade
+          Impala to beta version 0.6. If you upgrade Impala to beta version 0.6, you must upgrade Cloudera Manager
+          to 4.5.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="incompatible_changes_04">
+
+    <title>Incompatible Change Introduced in Version 0.4 of the Impala Beta Release</title>
+
+    <conbody>
+
+      <ul>
+        <li>
+          Cloudera Manager 4.1.3 supports only version 0.4 of the Impala Beta Release. It does not support
+          the earlier beta versions. If you upgrade your Cloudera Manager installation, you must also upgrade
+          Impala to beta version 0.4. If you upgrade Impala to beta version 0.4, you must upgrade Cloudera Manager
+          to 4.1.3.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+
+  <concept id="incompatible_changes_03">
+
+    <title>Incompatible Change Introduced in Version 0.3 of the Impala Beta Release</title>
+
+    <conbody>
+
+      <ul>
+        <li>
+          Cloudera Manager 4.1.2 supports only version 0.3 of the Impala Beta Release. It does not support
+          the earlier beta versions. If you upgrade your Cloudera Manager installation, you must also upgrade
+          Impala to beta version 0.3. If you upgrade Impala to beta version 0.3, you must upgrade Cloudera Manager
+          to 4.1.2.
+        </li>
+      </ul>
+    </conbody>
+  </concept>
+</concept>


[08/51] [partial] incubator-impala git commit: IMPALA-3398: Add docs to main Impala branch.

Posted by jb...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_schedule_random_replica.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_schedule_random_replica.xml b/docs/topics/impala_schedule_random_replica.xml
new file mode 100644
index 0000000..39092e8
--- /dev/null
+++ b/docs/topics/impala_schedule_random_replica.xml
@@ -0,0 +1,80 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="schedule_random_replica" rev="2.5.0">
+
+  <title>SCHEDULE_RANDOM_REPLICA Query Option (<keyword keyref="impala25"/> or higher only)</title>
+  <titlealts audience="PDF"><navtitle>SCHEDULE_RANDOM_REPLICA</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.5.0">
+      <indexterm audience="Cloudera">SCHEDULE_RANDOM_REPLICA query option</indexterm>
+    </p>
+
+    <p>
+      The <codeph>SCHEDULE_RANDOM_REPLICA</codeph> query option fine-tunes the algorithm for deciding which host
+      processes each HDFS data block. It only applies to tables and partitions that are not enabled
+      for the HDFS caching feature.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/type_boolean"/>
+
+    <p conref="../shared/impala_common.xml#common/default_false"/>
+
+    <p conref="../shared/impala_common.xml#common/added_in_250"/>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      In the presence of HDFS cached replicas, Impala randomizes
+      which host processes each cached data block.
+      To ensure that HDFS data blocks are cached on more
+      than one host, use the <codeph>WITH REPLICATION</codeph> clause along with
+      the <codeph>CACHED IN</codeph> clause in a
+      <codeph>CREATE TABLE</codeph> or <codeph>ALTER TABLE</codeph> statement.
+      Specify a replication value greater than or equal to the HDFS block replication factor.
+    </p>
+
+    <p>
+      The <codeph>SCHEDULE_RANDOM_REPLICA</codeph> query option applies to tables and partitions
+      that <i>do not</i> use HDFS caching.
+      By default, Impala estimates how much work each host has done for
+      the query, and selects the host that has the lowest workload.
+      This algorithm is intended to reduce CPU hotspots arising when the
+      same host is selected to process multiple data blocks, but hotspots
+      might still arise for some combinations of queries and data layout.
+      When the <codeph>SCHEDULE_RANDOM_REPLICA</codeph> option is enabled,
+      Impala further randomizes the scheduling algorithm for non-HDFS cached blocks,
+      which can further reduce the chance of CPU hotspots.
+    </p>
+
+    <p rev="CDH-43739 IMPALA-2979">
+      This query option works in conjunction with the work scheduling improvements
+      in <keyword keyref="impala25_full"/> and higher. The scheduling improvements
+      distribute the processing for cached HDFS data blocks to minimize hotspots:
+      if a data block is cached on more than one host, Impala chooses which host
+      to process each block based on which host has read the fewest bytes during
+      the current query. Enable <codeph>SCHEDULE_RANDOM_REPLICA</codeph> setting if CPU hotspots
+      still persist because of cases where hosts are <q>tied</q> in terms of
+      the amount of work done; by default, Impala picks the first eligible host
+      in this case.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+    <p>
+      <xref href="impala_perf_hdfs_caching.xml#hdfs_caching"/>,
+      <xref href="impala_scalability.xml#scalability_hotspots"/>
+      , <xref href="impala_replica_preference.xml#replica_preference"/>
+    </p>
+
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_schema_design.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_schema_design.xml b/docs/topics/impala_schema_design.xml
new file mode 100644
index 0000000..4d08de5
--- /dev/null
+++ b/docs/topics/impala_schema_design.xml
@@ -0,0 +1,222 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="schema_design">
+
+  <title>Guidelines for Designing Impala Schemas</title>
+  <titlealts audience="PDF"><navtitle>Designing Schemas</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Planning"/>
+      <data name="Category" value="Sectionated Pages"/>
+      <data name="Category" value="Proof of Concept"/>
+      <data name="Category" value="Checklists"/>
+      <data name="Category" value="Guidelines"/>
+      <data name="Category" value="Best Practices"/>
+      <data name="Category" value="Performance"/>
+      <data name="Category" value="Compression"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Schemas"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Porting"/>
+      <data name="Category" value="Proof of Concept"/>
+      <data name="Category" value="Administrators"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      The guidelines in this topic help you to construct an optimized and scalable schema, one that integrates well
+      with your existing data management processes. Use these guidelines as a checklist when doing any
+      proof-of-concept work, porting exercise, or before deploying to production.
+    </p>
+
+    <p>
+      If you are adapting an existing database or Hive schema for use with Impala, read the guidelines in this
+      section and then see <xref href="impala_porting.xml#porting"/> for specific porting and compatibility tips.
+    </p>
+
+    <p outputclass="toc inpage"/>
+
+    <section id="schema_design_text_vs_binary">
+
+      <title>Prefer binary file formats over text-based formats.</title>
+
+      <p>
+        To save space and improve memory usage and query performance, use binary file formats for any large or
+        intensively queried tables. Parquet file format is the most efficient for data warehouse-style analytic
+        queries. Avro is the other binary file format that Impala supports, that you might already have as part of
+        a Hadoop ETL pipeline.
+      </p>
+
+      <p>
+        Although Impala can create and query tables with the RCFile and SequenceFile file formats, such tables are
+        relatively bulky due to the text-based nature of those formats, and are not optimized for data
+        warehouse-style queries due to their row-oriented layout. Impala does not support <codeph>INSERT</codeph>
+        operations for tables with these file formats.
+      </p>
+
+      <p>
+        Guidelines:
+      </p>
+
+      <ul>
+        <li>
+          For an efficient and scalable format for large, performance-critical tables, use the Parquet file format.
+        </li>
+
+        <li>
+          To deliver intermediate data during the ETL process, in a format that can also be used by other Hadoop
+          components, Avro is a reasonable choice.
+        </li>
+
+        <li>
+          For convenient import of raw data, use a text table instead of RCFile or SequenceFile, and convert to
+          Parquet in a later stage of the ETL process.
+        </li>
+      </ul>
+    </section>
+
+    <section id="schema_design_compression">
+
+      <title>Use Snappy compression where practical.</title>
+
+      <p>
+        Snappy compression involves low CPU overhead to decompress, while still providing substantial space
+        savings. In cases where you have a choice of compression codecs, such as with the Parquet and Avro file
+        formats, use Snappy compression unless you find a compelling reason to use a different codec.
+      </p>
+    </section>
+
+    <section id="schema_design_numeric_types">
+
+      <title>Prefer numeric types over strings.</title>
+
+      <p>
+        If you have numeric values that you could treat as either strings or numbers (such as
+        <codeph>YEAR</codeph>, <codeph>MONTH</codeph>, and <codeph>DAY</codeph> for partition key columns), define
+        them as the smallest applicable integer types. For example, <codeph>YEAR</codeph> can be
+        <codeph>SMALLINT</codeph>, <codeph>MONTH</codeph> and <codeph>DAY</codeph> can be <codeph>TINYINT</codeph>.
+        Although you might not see any difference in the way partitioned tables or text files are laid out on disk,
+        using numeric types will save space in binary formats such as Parquet, and in memory when doing queries,
+        particularly resource-intensive queries such as joins.
+      </p>
+    </section>
+
+<!-- Alan suggests not making this recommendation.
+<section id="schema_design_decimal">
+<title>Prefer DECIMAL types over FLOAT and DOUBLE.</title>
+<p>
+</p>
+</section>
+-->
+
+    <section id="schema_design_partitioning">
+
+      <title>Partition, but do not over-partition.</title>
+
+      <p>
+        Partitioning is an important aspect of performance tuning for Impala. Follow the procedures in
+        <xref href="impala_partitioning.xml#partitioning"/> to set up partitioning for your biggest, most
+        intensively queried tables.
+      </p>
+
+      <p>
+        If you are moving to Impala from a traditional database system, or just getting started in the Big Data
+        field, you might not have enough data volume to take advantage of Impala parallel queries with your
+        existing partitioning scheme. For example, if you have only a few tens of megabytes of data per day,
+        partitioning by <codeph>YEAR</codeph>, <codeph>MONTH</codeph>, and <codeph>DAY</codeph> columns might be
+        too granular. Most of your cluster might be sitting idle during queries that target a single day, or each
+        node might have very little work to do. Consider reducing the number of partition key columns so that each
+        partition directory contains several gigabytes worth of data.
+      </p>
+
+      <p rev="parquet_block_size">
+        For example, consider a Parquet table where each data file is 1 HDFS block, with a maximum block size of 1
+        GB. (In Impala 2.0 and later, the default Parquet block size is reduced to 256 MB. For this exercise, let's
+        assume you have bumped the size back up to 1 GB by setting the query option
+        <codeph>PARQUET_FILE_SIZE=1g</codeph>.) if you have a 10-node cluster, you need 10 data files (up to 10 GB)
+        to give each node some work to do for a query. But each core on each machine can process a separate data
+        block in parallel. With 16-core machines on a 10-node cluster, a query could process up to 160 GB fully in
+        parallel. If there are only a few data files per partition, not only are most cluster nodes sitting idle
+        during queries, so are most cores on those machines.
+      </p>
+
+      <p>
+        You can reduce the Parquet block size to as low as 128 MB or 64 MB to increase the number of files per
+        partition and improve parallelism. But also consider reducing the level of partitioning so that analytic
+        queries have enough data to work with.
+      </p>
+    </section>
+
+    <section id="schema_design_compute_stats">
+
+      <title>Always compute stats after loading data.</title>
+
+      <p>
+        Impala makes extensive use of statistics about data in the overall table and in each column, to help plan
+        resource-intensive operations such as join queries and inserting into partitioned Parquet tables. Because
+        this information is only available after data is loaded, run the <codeph>COMPUTE STATS</codeph> statement
+        on a table after loading or replacing data in a table or partition.
+      </p>
+
+      <p>
+        Having accurate statistics can make the difference between a successful operation, or one that fails due to
+        an out-of-memory error or a timeout. When you encounter performance or capacity issues, always use the
+        <codeph>SHOW STATS</codeph> statement to check if the statistics are present and up-to-date for all tables
+        in the query.
+      </p>
+
+      <p>
+        When doing a join query, Impala consults the statistics for each joined table to determine their relative
+        sizes and to estimate the number of rows produced in each join stage. When doing an <codeph>INSERT</codeph>
+        into a Parquet table, Impala consults the statistics for the source table to determine how to distribute
+        the work of constructing the data files for each partition.
+      </p>
+
+      <p>
+        See <xref href="impala_compute_stats.xml#compute_stats"/> for the syntax of the <codeph>COMPUTE
+        STATS</codeph> statement, and <xref href="impala_perf_stats.xml#perf_stats"/> for all the performance
+        considerations for table and column statistics.
+      </p>
+    </section>
+
+    <section id="schema_design_explain">
+
+      <title>Verify sensible execution plans with EXPLAIN and SUMMARY.</title>
+
+      <p>
+        Before executing a resource-intensive query, use the <codeph>EXPLAIN</codeph> statement to get an overview
+        of how Impala intends to parallelize the query and distribute the work. If you see that the query plan is
+        inefficient, you can take tuning steps such as changing file formats, using partitioned tables, running the
+        <codeph>COMPUTE STATS</codeph> statement, or adding query hints. For information about all of these
+        techniques, see <xref href="impala_performance.xml#performance"/>.
+      </p>
+
+      <p>
+        After you run a query, you can see performance-related information about how it actually ran by issuing the
+        <codeph>SUMMARY</codeph> command in <cmdname>impala-shell</cmdname>. Prior to Impala 1.4, you would use
+        the <codeph>PROFILE</codeph> command, but its highly technical output was only useful for the most
+        experienced users. <codeph>SUMMARY</codeph>, new in Impala 1.4, summarizes the most useful information for
+        all stages of execution, for all nodes rather than splitting out figures for each node.
+      </p>
+    </section>
+
+<!--
+<section id="schema_design_mem_limits">
+<title>Allocate resources Between Impala and batch jobs (MapReduce, Hive, Pig).</title>
+<p>
+</p>
+</section>
+
+<section id="schema_design_cm">
+<title>Use Cloudera Manager to monitor queries and overall performance.</title>
+<p>
+</p>
+</section>
+-->
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_schema_objects.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_schema_objects.xml b/docs/topics/impala_schema_objects.xml
new file mode 100644
index 0000000..a9ddfcc
--- /dev/null
+++ b/docs/topics/impala_schema_objects.xml
@@ -0,0 +1,57 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="schema_objects">
+
+  <title>Impala Schema Objects and Object Names</title>
+  <titlealts audience="PDF"><navtitle>Schema Objects and Object Names</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Schemas"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">schema objects</indexterm>
+      With Impala, you work with schema objects that are familiar to database users: primarily databases, tables, views,
+      and functions. The SQL syntax to work with these objects is explained in
+      <xref href="impala_langref_sql.xml#langref_sql"/>. This section explains the conceptual knowledge you need to
+      work with these objects and the various ways to specify their names.
+    </p>
+
+    <p>
+      Within a table, partitions can also be considered a kind of object. Partitioning is an important subject for
+      Impala, with its own documentation section covering use cases and performance considerations. See
+      <xref href="impala_partitioning.xml#partitioning"/> for details.
+    </p>
+
+    <p>
+      Impala does not have a counterpart of the <q>tablespace</q> notion from some database systems. By default,
+      all the data files for a database, table, or partition are located within nested folders within the HDFS file
+      system. You can also specify a particular HDFS location for a given Impala table or partition. The raw data
+      for these objects is represented as a collection of data files, providing the flexibility to load data by
+      simply moving files into the expected HDFS location.
+    </p>
+
+    <p>
+      Information about the schema objects is held in the
+      <xref href="impala_hadoop.xml#intro_metastore">metastore</xref> database. This database is shared between
+      Impala and Hive, allowing each to create, drop, and query each other's databases, tables, and so on. When
+      Impala makes a change to schema objects through a <codeph>CREATE</codeph>, <codeph>ALTER</codeph>,
+      <codeph>DROP</codeph>, <codeph>INSERT</codeph>, or <codeph>LOAD DATA</codeph> statement, it broadcasts those
+      changes to all nodes in the cluster through the <xref href="impala_components.xml#intro_catalogd">catalog
+      service</xref>. When you make such changes through Hive or directly through manipulating HDFS files, you use
+      the <xref href="impala_refresh.xml#refresh">REFRESH</xref> or
+      <xref href="impala_invalidate_metadata.xml#invalidate_metadata">INVALIDATE METADATA</xref> statements on the
+      Impala side to recognize the newly loaded data, new tables, and so on.
+    </p>
+
+    <p outputclass="toc"/>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_security.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_security.xml b/docs/topics/impala_security.xml
new file mode 100644
index 0000000..bf4606d
--- /dev/null
+++ b/docs/topics/impala_security.xml
@@ -0,0 +1,123 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="security">
+
+  <title><ph audience="standalone">Impala Security</ph><ph audience="integrated">Overview of Impala Security</ph></title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Concepts"/>
+      <data name="Category" value="Auditing"/>
+      <data name="Category" value="Governance"/>
+      <data name="Category" value="Authentication"/>
+      <data name="Category" value="Authorization"/>
+      <data name="Category" value="Administrators"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Impala includes a fine-grained authorization framework for Hadoop, based on the Sentry
+      open source project. Sentry authorization was added in Impala 1.1.0. Together with the Kerberos authentication
+      framework, Sentry takes Hadoop security to a new level needed for the requirements of highly regulated industries
+      such as healthcare, financial services, and government. Impala also includes
+      an auditing capability; Impala generates the audit data, the Cloudera Navigator product consolidates
+      the audit data from all nodes in the cluster, and Cloudera Manager lets you filter, visualize, and produce
+      reports. The auditing feature was added in Impala 1.1.1.
+    </p>
+
+    <p>
+      The Impala security features have several objectives. At the most basic level, security prevents
+      accidents or mistakes that could disrupt application processing, delete or corrupt data, or reveal data to
+      unauthorized users. More advanced security features and practices can harden the system against malicious
+      users trying to gain unauthorized access or perform other disallowed operations. The auditing feature
+      provides a way to confirm that no unauthorized access occurred, and detect whether any such attempts were
+      made. This is a critical set of features for production deployments in large organizations that handle
+      important or sensitive data. It sets the stage for multi-tenancy, where multiple applications run
+      concurrently and are prevented from interfering with each other.
+    </p>
+
+    <p>
+      The material in this section presumes that you are already familiar with administering secure Linux systems.
+      That is, you should know the general security practices for Linux and Hadoop, and their associated commands
+      and configuration files. For example, you should know how to create Linux users and groups, manage Linux
+      group membership, set Linux and HDFS file permissions and ownership, and designate the default permissions
+      and ownership for new files. You should be familiar with the configuration of the nodes in your Hadoop
+      cluster, and know how to apply configuration changes or run a set of commands across all the nodes.
+    </p>
+
+    <p>
+      The security features are divided into these broad categories:
+    </p>
+
+    <dl>
+      <dlentry>
+
+        <dt>
+          authorization
+        </dt>
+
+        <dd>
+          Which users are allowed to access which resources, and what operations are they allowed to perform?
+          Impala relies on the open source Sentry project for authorization. By default (when authorization is not
+          enabled), Impala does all read and write operations with the privileges of the <codeph>impala</codeph>
+          user, which is suitable for a development/test environment but not for a secure production environment.
+          When authorization is enabled, Impala uses the OS user ID of the user who runs
+          <cmdname>impala-shell</cmdname> or other client program, and associates various privileges with each
+          user. See <xref href="impala_authorization.xml#authorization"/> for details about setting up and managing
+          authorization.
+        </dd>
+
+      </dlentry>
+
+      <dlentry>
+
+        <dt>
+          authentication
+        </dt>
+
+        <dd>
+          How does Impala verify the identity of the user to confirm that they really are allowed to exercise the
+          privileges assigned to that user? Impala relies on the Kerberos subsystem for authentication. See
+          <xref href="impala_kerberos.xml#kerberos"/> for details about setting up and managing authentication.
+        </dd>
+
+      </dlentry>
+
+      <dlentry>
+
+        <dt>
+          auditing
+        </dt>
+
+        <dd>
+          What operations were attempted, and did they succeed or not? This feature provides a way to look back and
+          diagnose whether attempts were made to perform unauthorized operations. You use this information to track
+          down suspicious activity, and to see where changes are needed in authorization policies. The audit data
+          produced by this feature is collected by the Cloudera Manager product and then presented in a
+          user-friendly form by the Cloudera Manager product. See <xref href="impala_auditing.xml#auditing"/> for
+          details about setting up and managing auditing.
+        </dd>
+
+      </dlentry>
+    </dl>
+
+    <p outputclass="toc"/>
+
+    <p audience="integrated">
+      These other topics in the <cite>Security Guide</cite> cover how Impala integrates with security frameworks
+      such as Kerberos, LDAP, and Sentry:
+      <ul>
+        <li>
+          <xref href="impala_authentication.xml#authentication"/>
+        </li>
+
+        <li>
+          <xref href="impala_authorization.xml#authorization"/>
+        </li>
+      </ul>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_security_files.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_security_files.xml b/docs/topics/impala_security_files.xml
new file mode 100644
index 0000000..befe696
--- /dev/null
+++ b/docs/topics/impala_security_files.xml
@@ -0,0 +1,67 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="secure_files">
+
+  <title>Securing Impala Data and Log Files</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="Logs"/>
+      <data name="Category" value="HDFS"/>
+      <data name="Category" value="Administrators"/>
+      <!-- To do for John: mention redaction as a fallback to keep sensitive info out of the log files. -->
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      One aspect of security is to protect files from unauthorized access at the filesystem level. For example, if
+      you store sensitive data in HDFS, you specify permissions on the associated files and directories in HDFS to
+      restrict read and write permissions to the appropriate users and groups.
+    </p>
+
+    <p>
+      If you issue queries containing sensitive values in the <codeph>WHERE</codeph> clause, such as financial
+      account numbers, those values are stored in Impala log files in the Linux filesystem and you must secure
+      those files also. For the locations of Impala log files, see <xref href="impala_logging.xml#logging"/>.
+    </p>
+
+    <p>
+      All Impala read and write operations are performed under the filesystem privileges of the
+      <codeph>impala</codeph> user. The <codeph>impala</codeph> user must be able to read all directories and data
+      files that you query, and write into all the directories and data files for <codeph>INSERT</codeph> and
+      <codeph>LOAD DATA</codeph> statements. At a minimum, make sure the <codeph>impala</codeph> user is in the
+      <codeph>hive</codeph> group so that it can access files and directories shared between Impala and Hive. See
+      <xref href="impala_prereqs.xml#prereqs_account"/> for more details.
+    </p>
+
+    <p>
+      Setting file permissions is necessary for Impala to function correctly, but is not an effective security
+      practice by itself:
+    </p>
+
+    <ul>
+      <li>
+      <p>
+        The way to ensure that only authorized users can submit requests for databases and tables they are allowed
+        to access is to set up Sentry authorization, as explained in
+        <xref href="impala_authorization.xml#authorization"/>. With authorization enabled, the checking of the user
+        ID and group is done by Impala, and unauthorized access is blocked by Impala itself. The actual low-level
+        read and write requests are still done by the <codeph>impala</codeph> user, so you must have appropriate
+        file and directory permissions for that user ID.
+      </p>
+      </li>
+
+      <li>
+      <p>
+        You must also set up Kerberos authentication, as described in <xref href="impala_kerberos.xml#kerberos"/>,
+        so that users can only connect from trusted hosts. With Kerberos enabled, if someone connects a new host to
+        the network and creates user IDs that match your privileged IDs, they will be blocked from connecting to
+        Impala at all from that host.
+      </p>
+      </li>
+    </ul>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_security_guidelines.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_security_guidelines.xml b/docs/topics/impala_security_guidelines.xml
new file mode 100644
index 0000000..e7713ff
--- /dev/null
+++ b/docs/topics/impala_security_guidelines.xml
@@ -0,0 +1,108 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.1" id="security_guidelines">
+
+  <title>Security Guidelines for Impala</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Planning"/>
+      <data name="Category" value="Guidelines"/>
+      <data name="Category" value="Best Practices"/>
+      <data name="Category" value="Administrators"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      The following are the major steps to harden a cluster running Impala against accidents and mistakes, or
+      malicious attackers trying to access sensitive data:
+    </p>
+
+    <ul>
+      <li>
+      <p>
+        Secure the <codeph>root</codeph> account. The <codeph>root</codeph> user can tamper with the
+        <cmdname>impalad</cmdname> daemon, read and write the data files in HDFS, log into other user accounts, and
+        access other system services that are beyond the control of Impala.
+      </p>
+      </li>
+
+      <li>
+      <p>
+        Restrict membership in the <codeph>sudoers</codeph> list (in the <filepath>/etc/sudoers</filepath> file).
+        The users who can run the <codeph>sudo</codeph> command can do many of the same things as the
+        <codeph>root</codeph> user.
+      </p>
+      </li>
+
+      <li>
+      <p>
+        Ensure the Hadoop ownership and permissions for Impala data files are restricted.
+      </p>
+      </li>
+
+      <li>
+      <p>
+        Ensure the Hadoop ownership and permissions for Impala log files are restricted.
+      </p>
+      </li>
+
+      <li>
+      <p>
+        Ensure that the Impala web UI (available by default on port 25000 on each Impala node) is
+        password-protected. See <xref href="impala_webui.xml#webui"/> for details.
+      </p>
+      </li>
+
+      <li>
+      <p>
+        Create a policy file that specifies which Impala privileges are available to users in particular Hadoop
+        groups (which by default map to Linux OS groups). Create the associated Linux groups using the
+        <cmdname>groupadd</cmdname> command if necessary.
+      </p>
+      </li>
+
+      <li>
+      <p>
+        The Impala authorization feature makes use of the HDFS file ownership and permissions mechanism; for
+        background information, see the
+        <xref href="https://archive.cloudera.com/cdh/3/hadoop/hdfs_permissions_guide.html" scope="external" format="html">CDH
+        HDFS Permissions Guide</xref>. Set up users and assign them to groups at the OS level, corresponding to the
+        different categories of users with different access levels for various databases, tables, and HDFS
+        locations (URIs). Create the associated Linux users using the <cmdname>useradd</cmdname> command if
+        necessary, and add them to the appropriate groups with the <cmdname>usermod</cmdname> command.
+      </p>
+      </li>
+
+      <li>
+      <p>
+        Design your databases, tables, and views with database and table structure to allow policy rules to specify
+        simple, consistent rules. For example, if all tables related to an application are inside a single
+        database, you can assign privileges for that database and use the <codeph>*</codeph> wildcard for the table
+        name. If you are creating views with different privileges than the underlying base tables, you might put
+        the views in a separate database so that you can use the <codeph>*</codeph> wildcard for the database
+        containing the base tables, while specifying the precise names of the individual views. (For specifying
+        table or database names, you either specify the exact name or <codeph>*</codeph> to mean all the databases
+        on a server, or all the tables and views in a database.)
+      </p>
+      </li>
+
+      <li>
+      <p>
+        Enable authorization by running the <codeph>impalad</codeph> daemons with the <codeph>-server_name</codeph>
+        and <codeph>-authorization_policy_file</codeph> options on all nodes. (The authorization feature does not
+        apply to the <cmdname>statestored</cmdname> daemon, which has no access to schema objects or data files.)
+      </p>
+      </li>
+
+      <li>
+      <p>
+        Set up authentication using Kerberos, to make sure users really are who they say they are.
+      </p>
+      </li>
+    </ul>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_security_install.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_security_install.xml b/docs/topics/impala_security_install.xml
new file mode 100644
index 0000000..56d34bc
--- /dev/null
+++ b/docs/topics/impala_security_install.xml
@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="security_install">
+
+  <title>Installation Considerations for Impala Security</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Administrators"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      Impala 1.1 comes set up with all the software and settings needed to enable security when you run the
+      <cmdname>impalad</cmdname> daemon with the new security-related options (<codeph>-server_name</codeph> and
+      <codeph>-authorization_policy_file</codeph>). You do not need to change any environment variables or install
+      any additional JAR files. In a cluster managed by Cloudera Manager, you do not need to change any settings in
+      Cloudera Manager.
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_security_metastore.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_security_metastore.xml b/docs/topics/impala_security_metastore.xml
new file mode 100644
index 0000000..246333f
--- /dev/null
+++ b/docs/topics/impala_security_metastore.xml
@@ -0,0 +1,40 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="security_metastore">
+
+  <title>Securing the Hive Metastore Database</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Hive"/>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="Metastore"/>
+      <data name="Category" value="Databases"/>
+      <data name="Category" value="Administrators"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+<!-- Some of this copied from earlier. Split out both instances into conrefs. -->
+
+    <p>
+      It is important to secure the Hive metastore, so that users cannot access the names or other information
+      about databases and tables the through the Hive client or by querying the metastore database. Do this by
+      turning on Hive metastore security, using the instructions in the
+      <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_hive_security.html" scope="external" format="html">CDH 5 Security Guide</xref>
+      for securing different Hive components:
+    </p>
+
+    <ul>
+      <li>
+        Secure the Hive Metastore.
+      </li>
+
+      <li>
+        In addition, allow access to the metastore only from the HiveServer2 server, and then disable local access
+        to the HiveServer2 server.
+      </li>
+    </ul>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_security_webui.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_security_webui.xml b/docs/topics/impala_security_webui.xml
new file mode 100644
index 0000000..7ebd2ef
--- /dev/null
+++ b/docs/topics/impala_security_webui.xml
@@ -0,0 +1,66 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="1.1" id="security_webui">
+
+  <title>Securing the Impala Web User Interface</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Troubleshooting"/>
+      <data name="Category" value="Security"/>
+      <data name="Category" value="Administrators"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      The instructions in this section presume you are familiar with the
+      <xref href="http://en.wikipedia.org/wiki/.htpasswd" scope="external" format="html">
+      <filepath>.htpasswd</filepath> mechanism</xref> commonly used to password-protect pages on web servers.
+    </p>
+
+    <p>
+      Password-protect the Impala web UI that listens on port 25000 by default. Set up a
+      <filepath>.htpasswd</filepath> file in the <codeph>$IMPALA_HOME</codeph> directory, or start both the
+      <cmdname>impalad</cmdname> and <cmdname>statestored</cmdname> daemons with the
+      <codeph>--webserver_password_file</codeph> option to specify a different location (including the filename).
+    </p>
+
+    <p>
+      This file should only be readable by the Impala process and machine administrators, because it contains
+      (hashed) versions of passwords. The username / password pairs are not derived from Unix usernames, Kerberos
+      users, or any other system. The <codeph>domain</codeph> field in the password file must match the domain
+      supplied to Impala by the new command-line option <codeph>--webserver_authentication_domain</codeph>. The
+      default is <codeph>mydomain.com</codeph>.
+<!-- Password generator cited by Henry: <xref href="http://www.askapache.com/online-tools/htpasswd-generator/" scope="external" format="html"/> -->
+    </p>
+
+    <p>
+      Impala also supports using HTTPS for secure web traffic. To do so, set
+      <codeph>--webserver_certificate_file</codeph> to refer to a valid <codeph>.pem</codeph> TLS/SSL certificate file.
+      Impala will automatically start using HTTPS once the TLS/SSL certificate has been read and validated. A
+      <codeph>.pem</codeph> file is basically a private key, followed by a signed TLS/SSL certificate; make sure to
+      concatenate both parts when constructing the <codeph>.pem</codeph> file.
+<!-- Certificate info cited by Henry: <xref href="http://www.akadia.com/services/ssh_test_certificate.html" scope="external" format="html"/>
+This page was very useful for creating a certificate and private key file;
+the last step which was missing was to append one file to the other to make the <codeph>.pem</codeph> file. -->
+    </p>
+
+    <p>
+      If Impala cannot find or parse the <codeph>.pem</codeph> file, it prints an error message and quits.
+    </p>
+
+    <note>
+      <p>
+        If the private key is encrypted using a passphrase, Impala will ask for that passphrase on startup, which
+        is not useful for a large cluster. In that case, remove the passphrase and make the <codeph>.pem</codeph>
+        file readable only by Impala and administrators.
+      </p>
+      <p>
+        When you turn on TLS/SSL for the Impala web UI, the associated URLs change from <codeph>http://</codeph>
+        prefixes to <codeph>https://</codeph>. Adjust any bookmarks or application code that refers to those URLs.
+      </p>
+    </note>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_select.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_select.xml b/docs/topics/impala_select.xml
new file mode 100644
index 0000000..321c371
--- /dev/null
+++ b/docs/topics/impala_select.xml
@@ -0,0 +1,206 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="select">
+
+  <title>SELECT Statement</title>
+  <titlealts audience="PDF"><navtitle>SELECT</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Reports"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+      <!-- This is such an important statement, think if there are more applicable categories. -->
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">SELECT statement</indexterm>
+      The <codeph>SELECT</codeph> statement performs queries, retrieving data from one or more tables and producing
+      result sets consisting of rows and columns.
+    </p>
+
+    <p>
+      The Impala <codeph><xref href="impala_insert.xml#insert">INSERT</xref></codeph> statement also typically ends
+      with a <codeph>SELECT</codeph> statement, to define data to copy from one table to another.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>[WITH <i>name</i> AS (<i>select_expression</i>) [, ...] ]
+SELECT
+  [ALL | DISTINCT]
+  [STRAIGHT_JOIN]
+  <i>expression</i> [, <i>expression</i> ...]
+FROM <i>table_reference</i> [, <i>table_reference</i> ...]
+[[FULL | [LEFT | RIGHT] INNER | [LEFT | RIGHT] OUTER | [LEFT | RIGHT] SEMI | [LEFT | RIGHT] ANTI | CROSS]
+  JOIN <i>table_reference</i>
+  [ON <i>join_equality_clauses</i> | USING (<varname>col1</varname>[, <varname>col2</varname> ...]] ...
+WHERE <i>conditions</i>
+GROUP BY { <i>column</i> | <i>expression</i> [ASC | DESC] [NULLS FIRST | NULLS LAST] [, ...] }
+HAVING <codeph>conditions</codeph>
+GROUP BY { <i>column</i> | <i>expression</i> [ASC | DESC] [, ...] }
+LIMIT <i>expression</i> [OFFSET <i>expression</i>]
+[UNION [ALL] <i>select_statement</i>] ...]
+</codeblock>
+
+    <p>
+      Impala <codeph>SELECT</codeph> queries support:
+    </p>
+
+    <ul>
+      <li>
+        SQL scalar data types: <codeph><xref href="impala_boolean.xml#boolean">BOOLEAN</xref></codeph>,
+        <codeph><xref href="impala_tinyint.xml#tinyint">TINYINT</xref></codeph>,
+        <codeph><xref href="impala_smallint.xml#smallint">SMALLINT</xref></codeph>,
+        <codeph><xref href="impala_int.xml#int">INT</xref></codeph>,
+        <codeph><xref href="impala_bigint.xml#bigint">BIGINT</xref></codeph>,
+        <codeph><xref href="impala_decimal.xml#decimal">DECIMAL</xref></codeph>
+        <codeph><xref href="impala_float.xml#float">FLOAT</xref></codeph>,
+        <codeph><xref href="impala_double.xml#double">DOUBLE</xref></codeph>,
+        <codeph><xref href="impala_timestamp.xml#timestamp">TIMESTAMP</xref></codeph>,
+        <codeph><xref href="impala_string.xml#string">STRING</xref></codeph>,
+        <codeph><xref href="impala_varchar.xml#varchar">VARCHAR</xref></codeph>,
+        <codeph><xref href="impala_char.xml#char">CHAR</xref></codeph>.
+      </li>
+
+<!-- To do: Consider promoting 'querying complex types' to its own subtopic or pseudo-heading. -->
+      <li rev="2.3.0">
+        The complex data types <codeph>ARRAY</codeph>, <codeph>STRUCT</codeph>, and <codeph>MAP</codeph>,
+        are available in <keyword keyref="impala23_full"/> and higher.
+        Queries involving these types typically involve special qualified names
+        using dot notation for referring to the complex column fields,
+        and join clauses for bringing the complex columns into the result set.
+        See <xref href="impala_complex_types.xml#complex_types"/> for details.
+      </li>
+
+      <li rev="1.1">
+        An optional <xref href="impala_with.xml#with"><codeph>WITH</codeph> clause</xref> before the
+        <codeph>SELECT</codeph> keyword, to define a subquery whose name or column names can be referenced from
+        later in the main query. This clause lets you abstract repeated clauses, such as aggregation functions,
+        that are referenced multiple times in the same query.
+      </li>
+
+      <li>
+        By default, one <codeph>DISTINCT</codeph> clause per query. See <xref href="impala_distinct.xml#distinct"/>
+        for details. See <xref href="impala_appx_count_distinct.xml#appx_count_distinct"/> for a query option to
+        allow multiple <codeph>COUNT(DISTINCT)</codeph> impressions in the same query.
+      </li>
+
+      <li>
+        Subqueries in a <codeph>FROM</codeph> clause. In <keyword keyref="impala20_full"/> and higher,
+        subqueries can also go in the <codeph>WHERE</codeph> clause, for example with the
+        <codeph>IN()</codeph>, <codeph>EXISTS</codeph>, and <codeph>NOT EXISTS</codeph> operators.
+      </li>
+
+      <li>
+        <codeph>WHERE</codeph>, <codeph>GROUP BY</codeph>, <codeph>HAVING</codeph> clauses.
+      </li>
+
+      <li rev="obwl">
+        <codeph><xref href="impala_order_by.xml#order_by">ORDER BY</xref></codeph>. Prior to Impala 1.4.0, Impala
+        required that queries using an <codeph>ORDER BY</codeph> clause also include a
+        <codeph><xref href="impala_limit.xml#limit">LIMIT</xref></codeph> clause. In Impala 1.4.0 and higher, this
+        restriction is lifted; sort operations that would exceed the Impala memory limit automatically use a
+        temporary disk work area to perform the sort.
+      </li>
+
+      <li>
+        <p conref="../shared/impala_common.xml#common/join_types"/>
+        <p>
+          See <xref href="impala_joins.xml#joins"/> for details and examples of join queries.
+        </p>
+      </li>
+
+      <li>
+        <codeph>UNION ALL</codeph>.
+      </li>
+
+      <li>
+        <codeph>LIMIT</codeph>.
+      </li>
+
+      <li>
+        External tables.
+      </li>
+
+      <li>
+        Relational operators such as greater than, less than, or equal to.
+      </li>
+
+      <li>
+        Arithmetic operators such as addition or subtraction.
+      </li>
+
+      <li>
+        Logical/Boolean operators <codeph>AND</codeph>, <codeph>OR</codeph>, and <codeph>NOT</codeph>. Impala does
+        not support the corresponding symbols <codeph>&amp;&amp;</codeph>, <codeph>||</codeph>, and
+        <codeph>!</codeph>.
+      </li>
+
+      <li>
+        Common SQL built-in functions such as <codeph>COUNT</codeph>, <codeph>SUM</codeph>, <codeph>CAST</codeph>,
+        <codeph>LIKE</codeph>, <codeph>IN</codeph>, <codeph>BETWEEN</codeph>, and <codeph>COALESCE</codeph>. Impala
+        specifically supports built-ins described in <xref href="impala_functions.xml#builtins"/>.
+      </li>
+    </ul>
+
+    <p conref="../shared/impala_common.xml#common/ignore_file_extensions"/>
+
+    <p conref="../shared/impala_common.xml#common/security_blurb"/>
+    <p conref="../shared/impala_common.xml#common/redaction_yes"/>
+
+    <p conref="../shared/impala_common.xml#common/s3_blurb"/>
+    <p conref="../shared/impala_common.xml#common/s3_block_splitting"/>
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_yes"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb"/>
+    <p rev="CDH-19187">
+      The user ID that the <cmdname>impalad</cmdname> daemon runs under,
+      typically the <codeph>impala</codeph> user, must have read
+      permissions for the files in all applicable directories in all source tables,
+      and read and execute permissions for the relevant data directories.
+      (A <codeph>SELECT</codeph> operation could read files from multiple different HDFS directories
+      if the source table is partitioned.)
+      If a query attempts to read a data file and is unable to because of an HDFS permission error,
+      the query halts and does not return any further results.
+    </p>
+
+    <p outputclass="toc"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      The <codeph>SELECT</codeph> syntax is so extensive that it forms its own category of statements: queries. The
+      other major classifications of SQL statements are data definition language (see
+      <xref href="impala_ddl.xml#ddl"/>) and data manipulation language (see <xref href="impala_dml.xml#dml"/>).
+    </p>
+
+    <p>
+      Because the focus of Impala is on fast queries with interactive response times over huge data sets, query
+      performance and scalability are important considerations. See
+      <xref href="impala_performance.xml#performance"/> and <xref href="impala_scalability.xml#scalability"/> for
+      details.
+    </p>
+  </conbody>
+
+  <concept id="where" audience="Cloudera">
+
+<!-- WHERE hidden for the moment until there's the chance to add some reasonably comprehensive content
+
+     and make it its own file. -->
+
+    <title>WHERE Clause</title>
+
+    <conbody>
+
+      <p/>
+    </conbody>
+  </concept>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_seq_compression_mode.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_seq_compression_mode.xml b/docs/topics/impala_seq_compression_mode.xml
new file mode 100644
index 0000000..cb7cb05
--- /dev/null
+++ b/docs/topics/impala_seq_compression_mode.xml
@@ -0,0 +1,29 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="seq_compression_mode" rev="2.5.0">
+
+<!-- This option is for internal use only and might go away without ever being documented. It's hidden in the DITA map. -->
+
+  <title>SEQ_COMPRESSION_MODE Query Option</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="Impala Query Options"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.5.0">
+      <indexterm audience="Cloudera">RM_INITIAL_MEM query option</indexterm>
+    </p>
+
+    <p>
+      <b>Type:</b>
+    </p>
+
+    <p>
+      <b>Default:</b>
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_seqfile.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_seqfile.xml b/docs/topics/impala_seqfile.xml
new file mode 100644
index 0000000..860007e
--- /dev/null
+++ b/docs/topics/impala_seqfile.xml
@@ -0,0 +1,239 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="seqfile">
+
+  <title id="sequencefile">Using the SequenceFile File Format with Impala Tables</title>
+  <titlealts audience="PDF"><navtitle>SequenceFile Data Files</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <!-- <data name="Category" value="SequenceFile"/> -->
+      <data name="Category" value="File Formats"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">SequenceFile support in Impala</indexterm>
+      Impala supports using SequenceFile data files.
+    </p>
+
+    <table>
+      <title>SequenceFile Format Support in Impala</title>
+      <tgroup cols="5">
+        <colspec colname="1" colwidth="10*"/>
+        <colspec colname="2" colwidth="10*"/>
+        <colspec colname="3" colwidth="20*"/>
+        <colspec colname="4" colwidth="30*"/>
+        <colspec colname="5" colwidth="30*"/>
+        <thead>
+          <row>
+            <entry>
+              File Type
+            </entry>
+            <entry>
+              Format
+            </entry>
+            <entry>
+              Compression Codecs
+            </entry>
+            <entry>
+              Impala Can CREATE?
+            </entry>
+            <entry>
+              Impala Can INSERT?
+            </entry>
+          </row>
+        </thead>
+        <tbody>
+          <row conref="impala_file_formats.xml#file_formats/sequencefile_support">
+            <entry/>
+          </row>
+        </tbody>
+      </tgroup>
+    </table>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="seqfile_create">
+
+    <title>Creating SequenceFile Tables and Loading Data</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="ETL"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        If you do not have an existing data file to use, begin by creating one in the appropriate format.
+      </p>
+
+      <p>
+        <b>To create a SequenceFile table:</b>
+      </p>
+
+      <p>
+        In the <codeph>impala-shell</codeph> interpreter, issue a command similar to:
+      </p>
+
+<codeblock>create table sequencefile_table (<varname>column_specs</varname>) stored as sequencefile;</codeblock>
+
+      <p>
+        Because Impala can query some kinds of tables that it cannot currently write to, after creating tables of
+        certain file formats, you might use the Hive shell to load the data. See
+        <xref href="impala_file_formats.xml#file_formats"/> for details. After loading data into a table through
+        Hive or other mechanism outside of Impala, issue a <codeph>REFRESH <varname>table_name</varname></codeph>
+        statement the next time you connect to the Impala node, before querying the table, to make Impala recognize
+        the new data.
+      </p>
+
+      <p>
+        For example, here is how you might create some SequenceFile tables in Impala (by specifying the columns
+        explicitly, or cloning the structure of another table), load data through Hive, and query them through
+        Impala:
+      </p>
+
+<codeblock>$ impala-shell -i localhost
+[localhost:21000] &gt; create table seqfile_table (x int) stored as sequencefile;
+[localhost:21000] &gt; create table seqfile_clone like some_other_table stored as sequencefile;
+[localhost:21000] &gt; quit;
+
+$ hive
+hive&gt; insert into table seqfile_table select x from some_other_table;
+3 Rows loaded to seqfile_table
+Time taken: 19.047 seconds
+hive&gt; quit;
+
+$ impala-shell -i localhost
+[localhost:21000] &gt; select * from seqfile_table;
+Returned 0 row(s) in 0.23s
+[localhost:21000] &gt; -- Make Impala recognize the data loaded through Hive;
+[localhost:21000] &gt; refresh seqfile_table;
+[localhost:21000] &gt; select * from seqfile_table;
++---+
+| x |
++---+
+| 1 |
+| 2 |
+| 3 |
++---+
+Returned 3 row(s) in 0.23s</codeblock>
+
+      <p conref="../shared/impala_common.xml#common/complex_types_unsupported_filetype"/>
+
+    </conbody>
+  </concept>
+
+  <concept id="seqfile_compression">
+
+    <title>Enabling Compression for SequenceFile Tables</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Compression"/>
+      <data name="Category" value="Snappy"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="Cloudera">compression</indexterm>
+        You may want to enable compression on existing tables. Enabling compression provides performance gains in
+        most cases and is supported for SequenceFile tables. For example, to enable Snappy compression, you would
+        specify the following additional settings when loading data through the Hive shell:
+      </p>
+
+<codeblock>hive&gt; SET hive.exec.compress.output=true;
+hive&gt; SET mapred.max.split.size=256000000;
+hive&gt; SET mapred.output.compression.type=BLOCK;
+hive&gt; SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
+hive&gt; insert overwrite table <varname>new_table</varname> select * from <varname>old_table</varname>;</codeblock>
+
+      <p>
+        If you are converting partitioned tables, you must complete additional steps. In such a case, specify
+        additional settings similar to the following:
+      </p>
+
+<codeblock>hive&gt; create table <varname>new_table</varname> (<varname>your_cols</varname>) partitioned by (<varname>partition_cols</varname>) stored as <varname>new_format</varname>;
+hive&gt; SET hive.exec.dynamic.partition.mode=nonstrict;
+hive&gt; SET hive.exec.dynamic.partition=true;
+hive&gt; insert overwrite table <varname>new_table</varname> partition(<varname>comma_separated_partition_cols</varname>) select * from <varname>old_table</varname>;</codeblock>
+
+      <p>
+        Remember that Hive does not require that you specify a source format for it. Consider the case of
+        converting a table with two partition columns called <codeph>year</codeph> and <codeph>month</codeph> to a
+        Snappy compressed SequenceFile. Combining the components outlined previously to complete this table
+        conversion, you would specify settings similar to the following:
+      </p>
+
+<codeblock>hive&gt; create table TBL_SEQ (int_col int, string_col string) STORED AS SEQUENCEFILE;
+hive&gt; SET hive.exec.compress.output=true;
+hive&gt; SET mapred.max.split.size=256000000;
+hive&gt; SET mapred.output.compression.type=BLOCK;
+hive&gt; SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
+hive&gt; SET hive.exec.dynamic.partition.mode=nonstrict;
+hive&gt; SET hive.exec.dynamic.partition=true;
+hive&gt; INSERT OVERWRITE TABLE tbl_seq SELECT * FROM tbl;</codeblock>
+
+      <p>
+        To complete a similar process for a table that includes partitions, you would specify settings similar to
+        the following:
+      </p>
+
+<codeblock>hive&gt; CREATE TABLE tbl_seq (int_col INT, string_col STRING) PARTITIONED BY (year INT) STORED AS SEQUENCEFILE;
+hive&gt; SET hive.exec.compress.output=true;
+hive&gt; SET mapred.max.split.size=256000000;
+hive&gt; SET mapred.output.compression.type=BLOCK;
+hive&gt; SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
+hive&gt; SET hive.exec.dynamic.partition.mode=nonstrict;
+hive&gt; SET hive.exec.dynamic.partition=true;
+hive&gt; INSERT OVERWRITE TABLE tbl_seq PARTITION(year) SELECT * FROM tbl;</codeblock>
+
+      <note>
+        <p>
+          The compression type is specified in the following command:
+        </p>
+<codeblock>SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;</codeblock>
+        <p>
+          You could elect to specify alternative codecs such as <codeph>GzipCodec</codeph> here.
+        </p>
+      </note>
+    </conbody>
+  </concept>
+
+  <concept audience="Cloudera" id="seqfile_data_types">
+
+    <title>Data Type Considerations for SequenceFile Tables</title>
+
+    <conbody>
+
+      <p></p>
+    </conbody>
+  </concept>
+
+  <concept id="seqfile_performance">
+
+    <title>Query Performance for Impala SequenceFile Tables</title>
+
+    <conbody>
+
+      <p>
+        In general, expect query performance with SequenceFile tables to be
+        faster than with tables using text data, but slower than with
+        Parquet tables. See <xref href="impala_parquet.xml#parquet"/>
+        for information about using the Parquet file format for
+        high-performance analytic queries.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/s3_block_splitting"/>
+
+    </conbody>
+  </concept>
+
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_set.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_set.xml b/docs/topics/impala_set.xml
new file mode 100644
index 0000000..f477269
--- /dev/null
+++ b/docs/topics/impala_set.xml
@@ -0,0 +1,192 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="2.0.0" id="set">
+
+  <title>SET Statement</title>
+  <titlealts audience="PDF"><navtitle>SET</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Querying"/>
+      <data name="Category" value="Configuring"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p rev="2.0.0">
+      <indexterm audience="Cloudera">SET statement</indexterm>
+      Specifies values for query options that control the runtime behavior of other statements within the same
+      session.
+    </p>
+
+    <p rev="2.5.0 IMPALA-2180">
+      In <keyword keyref="impala25_full"/> and higher, <codeph>SET</codeph> also defines user-specified substitution variables for
+      the <cmdname>impala-shell</cmdname> interpreter. This feature uses the <codeph>SET</codeph> command
+      built into <cmdname>impala-shell</cmdname> instead of the SQL <codeph>SET</codeph> statement.
+      Therefore the substitution mechanism only works with queries processed by <cmdname>impala-shell</cmdname>,
+      not with queries submitted through JDBC or ODBC.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/syntax_blurb"/>
+
+<codeblock>SET [<varname>query_option</varname>=<varname>option_value</varname>]
+</codeblock>
+
+    <p>
+      <codeph>SET</codeph> with no arguments returns a result set consisting of all available query options and
+      their current values.
+    </p>
+
+    <p>
+      The query option name and any string argument values are case-insensitive.
+    </p>
+
+    <p>
+      Each query option has a specific allowed notation for its arguments. Boolean options can be enabled and
+      disabled by assigning values of either <codeph>true</codeph> and <codeph>false</codeph>, or
+      <codeph>1</codeph> and <codeph>0</codeph>. Some numeric options accept a final character signifying the unit,
+      such as <codeph>2g</codeph> for 2 gigabytes or <codeph>100m</codeph> for 100 megabytes. See
+      <xref href="impala_query_options.xml#query_options"/> for the details of each query option.
+    </p>
+
+    <p>
+      <b>User-specified substitution variables:</b>
+    </p>
+
+    <p rev="2.5.0 IMPALA-2180">
+      In <keyword keyref="impala25_full"/> and higher, you can specify your own names and string substitution values
+      within the <cmdname>impala-shell</cmdname> interpreter. Once a substitution variable is set up,
+      its value is inserted into any SQL statement in that same <cmdname>impala-shell</cmdname> session
+      that contains the notation <codeph>${var:<varname>varname</varname>}</codeph>.
+      Using <codeph>SET</codeph> in an interactive <cmdname>impala-shell</cmdname> session overrides
+      any value for that same variable passed in through the <codeph>--var=<varname>varname</varname>=<varname>value</varname></codeph>
+      command-line option.
+    </p>
+
+    <p rev="2.5.0 IMPALA-2180">
+      For example, to set up some default parameters for report queries, but then override those default
+      within an <cmdname>impala-shell</cmdname> session, you might issue commands and statements such as
+      the following:
+    </p>
+
+<codeblock rev="2.5.0 IMPALA-2180">
+-- Initial setup for this example.
+create table staging_table (s string);
+insert into staging_table values ('foo'), ('bar'), ('bletch');
+
+create table production_table (s string);
+insert into production_table values ('North America'), ('EMEA'), ('Asia');
+quit;
+
+-- Start impala-shell with user-specified substitution variables,
+-- run a query, then override the variables with SET and run the query again.
+$ impala-shell --var=table_name=staging_table --var=cutoff=2
+... <varname>banner message</varname> ...
+[localhost:21000] > select s from ${var:table_name} order by s limit ${var:cutoff};
+Query: select s from staging_table order by s limit 2
++--------+
+| s      |
++--------+
+| bar    |
+| bletch |
++--------+
+Fetched 2 row(s) in 1.06s
+
+[localhost:21000] > set var:table_name=production_table;
+Variable TABLE_NAME set to production_table
+[localhost:21000] > set var:cutoff=3;
+Variable CUTOFF set to 3
+
+[localhost:21000] > select s from ${var:table_name} order by s limit ${var:cutoff};
+Query: select s from production_table order by s limit 3
++---------------+
+| s             |
++---------------+
+| Asia          |
+| EMEA          |
+| North America |
++---------------+
+</codeblock>
+
+    <p rev="2.5.0 IMPALA-2180">
+      The following example shows how <codeph>SET</codeph> with no parameters displays
+      all user-specified substitution variables, and how <codeph>UNSET</codeph> removes
+      the substitution variable entirely:
+    </p>
+
+<codeblock rev="2.5.0 IMPALA-2180">
+[localhost:21000] > set;
+Query options (defaults shown in []):
+  ABORT_ON_DEFAULT_LIMIT_EXCEEDED: [0]
+  ...
+  V_CPU_CORES: [0]
+
+Shell Options
+  LIVE_PROGRESS: False
+  LIVE_SUMMARY: False
+
+Variables:
+  CUTOFF: 3
+  TABLE_NAME: staging_table
+
+[localhost:21000] > unset var:cutoff;
+Unsetting variable CUTOFF
+[localhost:21000] > select s from ${var:table_name} order by s limit ${var:cutoff};
+Error: Unknown variable CUTOFF
+</codeblock>
+
+    <p rev="2.5.0 IMPALA-2180">
+      See <xref href="impala_shell_running_commands.xml"/> for more examples of using the
+      <codeph>--var</codeph>, <codeph>SET</codeph>, and <codeph>${var:<varname>varname</varname>}</codeph>
+      substitution technique in <cmdname>impala-shell</cmdname>.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+
+    <p>
+      <codeph>MEM_LIMIT</codeph> is probably the most commonly used query option. You can specify a high value to
+      allow a resource-intensive query to complete. For testing how queries would work on memory-constrained
+      systems, you might specify an artificially low value.
+    </p>
+
+    <p conref="../shared/impala_common.xml#common/complex_types_blurb"/>
+
+    <p conref="../shared/impala_common.xml#common/example_blurb"/>
+
+    <p>
+      The following example sets some numeric and some Boolean query options to control usage of memory, disk
+      space, and timeout periods, then runs a query whose success could depend on the options in effect:
+    </p>
+
+<codeblock>set mem_limit=64g;
+set DISABLE_UNSAFE_SPILLS=true;
+set parquet_file_size=400m;
+set RESERVATION_REQUEST_TIMEOUT=900000;
+insert overwrite parquet_table select c1, c2, count(c3) from text_table group by c1, c2, c3;
+</codeblock>
+
+    <p conref="../shared/impala_common.xml#common/added_in_20"/>
+
+    <p>
+      <codeph>SET</codeph> has always been available as an <cmdname>impala-shell</cmdname> command. Promoting it to
+      a SQL statement lets you use this feature in client applications through the JDBC and ODBC APIs.
+    </p>
+
+<!-- <p conref="../shared/impala_common.xml#common/jdbc_blurb"/> -->
+
+    <p conref="../shared/impala_common.xml#common/cancel_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/permissions_blurb_no"/>
+
+    <p conref="../shared/impala_common.xml#common/related_info"/>
+
+    <p>
+      See <xref href="impala_query_options.xml#query_options"/> for the query options you can adjust using this
+      statement.
+    </p>
+  </conbody>
+</concept>

http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/3be0f122/docs/topics/impala_shell_commands.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_shell_commands.xml b/docs/topics/impala_shell_commands.xml
new file mode 100644
index 0000000..885fd16
--- /dev/null
+++ b/docs/topics/impala_shell_commands.xml
@@ -0,0 +1,399 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="shell_commands">
+
+  <title>impala-shell Command Reference</title>
+  <titlealts audience="PDF"><navtitle>Command Reference</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <data name="Category" value="impala-shell"/>
+      <data name="Category" value="SQL"/>
+      <data name="Category" value="Data Analysts"/>
+      <data name="Category" value="Developers"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="Cloudera">impala-shell</indexterm>
+      Use the following commands within <codeph>impala-shell</codeph> to pass requests to the
+      <codeph>impalad</codeph> daemon that the shell is connected to. You can enter a command interactively at the
+      prompt, or pass it as the argument to the <codeph>-q</codeph> option of <codeph>impala-shell</codeph>. Most
+      of these commands are passed to the Impala daemon as SQL statements; refer to the corresponding
+      <xref href="impala_langref_sql.xml#langref_sql">SQL language reference sections</xref> for full syntax
+      details.
+    </p>
+
+    <table>
+      <tgroup cols="2">
+        <colspec colname="1" colwidth="10*"/>
+        <colspec colname="2" colwidth="40*"/>
+        <thead>
+          <row>
+            <entry>
+              Command
+            </entry>
+            <entry>
+              Explanation
+            </entry>
+          </row>
+        </thead>
+        <tbody>
+          <row id="alter_cmd">
+            <entry>
+              <p>
+                <codeph>alter</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Changes the underlying structure or settings of an Impala table, or a table shared between Impala
+                and Hive. See <xref href="impala_alter_table.xml#alter_table"/> and
+                <xref href="impala_alter_view.xml#alter_view"/> for details.
+              </p>
+            </entry>
+          </row>
+          <row rev="1.2.2" id="compute_cmd">
+            <entry>
+              <p>
+                <codeph>compute stats</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Gathers important performance-related information for a table, used by Impala to optimize queries.
+                See <xref href="impala_compute_stats.xml#compute_stats"/> for details.
+              </p>
+            </entry>
+          </row>
+          <row id="connect_cmd">
+            <entry>
+              <p>
+                <codeph>connect</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Connects to the specified instance of <codeph>impalad</codeph>. The default port of 21000 is
+                assumed unless you provide another value. You can connect to any host in your cluster that is
+                running <codeph>impalad</codeph>. If you connect to an instance of <codeph>impalad</codeph> that
+                was started with an alternate port specified by the <codeph>--fe_port</codeph> flag, you must
+                provide that alternate port. See <xref href="impala_connecting.xml#connecting"/> for examples.
+              </p>
+
+              <p conref="../shared/impala_common.xml#common/set_vs_connect"/>
+            </entry>
+          </row>
+          <row id="describe_cmd">
+            <entry>
+              <p>
+                <codeph>describe</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Shows the columns, column data types, and any column comments for a specified table.
+                <codeph>DESCRIBE FORMATTED</codeph> shows additional information such as the HDFS data directory,
+                partitions, and internal properties for the table. See <xref href="impala_describe.xml#describe"/>
+                for details about the basic <codeph>DESCRIBE</codeph> output and the <codeph>DESCRIBE
+                FORMATTED</codeph> variant. You can use <codeph>DESC</codeph> as shorthand for the
+                <codeph>DESCRIBE</codeph> command.
+              </p>
+            </entry>
+          </row>
+          <row id="drop_cmd">
+            <entry>
+              <p>
+                <codeph>drop</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Removes a schema object, and in some cases its associated data files. See
+                <xref href="impala_drop_table.xml#drop_table"/>, <xref href="impala_drop_view.xml#drop_view"/>,
+                <xref href="impala_drop_database.xml#drop_database"/>, and
+                <xref href="impala_drop_function.xml#drop_function"/> for details.
+              </p>
+            </entry>
+          </row>
+          <row id="explain_cmd">
+            <entry>
+              <p>
+                <codeph>explain</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Provides the execution plan for a query. <codeph>EXPLAIN</codeph> represents a query as a series of
+                steps. For example, these steps might be map/reduce stages, metastore operations, or file system
+                operations such as move or rename. See <xref href="impala_explain.xml#explain"/> and
+                <xref href="impala_explain_plan.xml#perf_explain"/> for details.
+              </p>
+            </entry>
+          </row>
+          <row id="help_cmd">
+            <entry>
+              <p>
+                <codeph>help</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Help provides a list of all available commands and options.
+              </p>
+            </entry>
+          </row>
+          <row id="history_cmd">
+            <entry>
+              <p>
+                <codeph>history</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Maintains an enumerated cross-session command history. This history is stored in the
+                <filepath>~/.impalahistory</filepath> file.
+              </p>
+            </entry>
+          </row>
+          <row id="insert_cmd">
+            <entry>
+              <p>
+                <codeph>insert</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Writes the results of a query to a specified table. This either overwrites table data or appends
+                data to the existing table content. See <xref href="impala_insert.xml#insert"/> for details.
+              </p>
+            </entry>
+          </row>
+          <row id="invalidate_metadata_cmd">
+            <entry>
+              <p>
+                <codeph>invalidate metadata</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Updates <cmdname>impalad</cmdname> metadata for table existence and structure. Use this command
+                after creating, dropping, or altering databases, tables, or partitions in Hive. See
+                <xref href="impala_invalidate_metadata.xml#invalidate_metadata"/> for details.
+              </p>
+            </entry>
+          </row>
+          <row id="profile_cmd">
+            <entry>
+              <p>
+                <codeph>profile</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Displays low-level information about the most recent query. Used for performance diagnosis and
+                tuning. <ph rev="1.4.0"> The report starts with the same information as produced by the
+                <codeph>EXPLAIN</codeph> statement and the <codeph>SUMMARY</codeph> command.</ph> See
+                <xref href="impala_explain_plan.xml#perf_profile"/> for details.
+              </p>
+            </entry>
+          </row>
+          <row id="quit_cmd">
+            <entry>
+              <p>
+                <codeph>quit</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Exits the shell. Remember to include the final semicolon so that the shell recognizes the end of
+                the command.
+              </p>
+            </entry>
+          </row>
+          <row id="refresh_cmd">
+            <entry>
+              <p>
+                <codeph>refresh</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Refreshes <cmdname>impalad</cmdname> metadata for the locations of HDFS blocks corresponding to
+                Impala data files. Use this command after loading new data files into an Impala table through Hive
+                or through HDFS commands. See <xref href="impala_refresh.xml#refresh"/> for details.
+              </p>
+            </entry>
+          </row>
+          <row id="select_cmd">
+            <entry>
+              <p>
+                <codeph>select</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Specifies the data set on which to complete some action. All information returned from
+                <codeph>select</codeph> can be sent to some output such as the console or a file or can be used to
+                complete some other element of query. See <xref href="impala_select.xml#select"/> for details.
+              </p>
+            </entry>
+          </row>
+          <row id="set_cmd">
+            <entry>
+              <p>
+                <codeph>set</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Manages query options for an <cmdname>impala-shell</cmdname> session. The available options are the
+                ones listed in <xref href="impala_query_options.xml#query_options"/>. These options are used for
+                query tuning and troubleshooting. Issue <codeph>SET</codeph> with no arguments to see the current
+                query options, either based on the <cmdname>impalad</cmdname> defaults, as specified by you at
+                <cmdname>impalad</cmdname> startup, or based on earlier <codeph>SET</codeph> statements in the same
+                session. To modify option values, issue commands with the syntax <codeph>set
+                <varname>option</varname>=<varname>value</varname></codeph>. To restore an option to its default,
+                use the <codeph>unset</codeph> command. Some options take Boolean values of <codeph>true</codeph>
+                and <codeph>false</codeph>. Others take numeric arguments, or quoted string values.
+              </p>
+
+              <p conref="../shared/impala_common.xml#common/set_vs_connect"/>
+
+              <p rev="2.0.0">
+                In Impala 2.0 and later, <codeph>SET</codeph> is available as a SQL statement for any kind of
+                application, not only through <cmdname>impala-shell</cmdname>. See
+                <xref href="impala_set.xml#set"/> for details.
+              </p>
+
+              <p rev="2.5.0 IMPALA-2180">
+                In Impala 2.5 and later, you can use <codeph>SET</codeph> to define your own substitution variables
+                within an <cmdname>impala-shell</cmdname> session.
+                Within a SQL statement, you substitute the value by using the notation <codeph>${var:<varname>variable_name</varname>}</codeph>.
+              </p>
+            </entry>
+          </row>
+          <row id="shell_cmd">
+            <entry>
+              <p>
+                <codeph>shell</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Executes the specified command in the operating system shell without exiting
+                <codeph>impala-shell</codeph>. You can use the <codeph>!</codeph> character as shorthand for the
+                <codeph>shell</codeph> command.
+              </p>
+
+              <note>
+                Quote any instances of the <codeph>--</codeph> or <codeph>/*</codeph> tokens to avoid them being
+                interpreted as the start of a comment. To embed comments within <codeph>source</codeph> or
+                <codeph>!</codeph> commands, use the shell comment character <codeph>#</codeph> before the comment
+                portion of the line.
+              </note>
+            </entry>
+          </row>
+          <row id="show_cmd">
+            <entry>
+              <p>
+                <codeph>show</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Displays metastore data for schema objects created and accessed through Impala, Hive, or both.
+                <codeph>show</codeph> can be used to gather information about objects such as databases, tables, and functions.
+                See <xref href="impala_show.xml#show"/> for details.
+              </p>
+            </entry>
+          </row>
+          <row id="source_cmd" rev="IMPALA-3397 CDH-40097">
+            <entry>
+              <p>
+                <codeph>source</codeph> or <codeph>src</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Executes one or more statements residing in a specified file from the local filesystem.
+                Allows you to perform the same kinds of batch operations as with the <codeph>-f</codeph> option,
+                but interactively within the interpreter. The file can contain SQL statements and other
+                <cmdname>impala-shell</cmdname> commands, including additional <codeph>SOURCE</codeph> commands
+                to perform a flexible sequence of actions. Each command or statement, except the last one in the file,
+                must end with a semicolon.
+                See <xref href="impala_shell_running_commands.xml#shell_running_commands"/> for examples.
+              </p>
+            </entry>
+          </row>
+          <row rev="1.4.0" id="summary_cmd">
+            <entry>
+              <p>
+                <codeph>summary</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Summarizes the work performed in various stages of a query. It provides a higher-level view of the
+                information displayed by the <codeph>EXPLAIN</codeph> command. Added in Impala 1.4.0. See
+                <xref href="impala_explain_plan.xml#perf_summary"/> for details about the report format
+                and how to interpret it.
+              </p>
+              <p rev="2.3.0">
+                In <keyword keyref="impala23_full"/> and higher, you can see a continuously updated report of
+                the summary information while a query is in progress.
+                See <xref href="impala_live_summary.xml#live_summary"/> for details.
+              </p>
+            </entry>
+          </row>
+          <row id="unset_cmd">
+            <entry>
+              <p>
+                <codeph>unset</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Removes any user-specified value for a query option and returns the option to its default value.
+                See <xref href="impala_query_options.xml#query_options"/> for the available query options.
+              </p>
+              <p rev="2.5.0 IMPALA-2180">
+                In <keyword keyref="impala25_full"/> and higher, it can also remove user-specified substitution variables
+                using the notation <codeph>UNSET VAR:<varname>variable_name</varname></codeph>.
+              </p>
+            </entry>
+          </row>
+          <row id="use_cmd">
+            <entry>
+              <p>
+                <codeph>use</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Indicates the database against which to execute subsequent commands. Lets you avoid using fully
+                qualified names when referring to tables in databases other than <codeph>default</codeph>. See
+                <xref href="impala_use.xml#use"/> for details. Not effective with the <codeph>-q</codeph> option,
+                because that option only allows a single statement in the argument.
+              </p>
+            </entry>
+          </row>
+          <row id="version_cmd">
+            <entry>
+              <p>
+                <codeph>version</codeph>
+              </p>
+            </entry>
+            <entry>
+              <p>
+                Returns Impala version information.
+              </p>
+            </entry>
+          </row>
+        </tbody>
+      </tgroup>
+    </table>
+  </conbody>
+</concept>