You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2018/06/07 00:24:20 UTC

[4/4] impala git commit: IMPALA-6714: [DOCS] ORC file format support

IMPALA-6714: [DOCS] ORC file format support

This document is wrote refering to RCFile and Parquet's docs.

The orc-support patch was merged in impala-2.12 and impala-3.0,
so we start to support ORC format as an experimental feature
since impala-2.12.

Change-Id: Ib1ee23ed844653c274babdce5a332dbe5c79b630
Reviewed-on: http://gerrit.cloudera.org:8080/10525
Reviewed-by: Balazs Jeszenszky <je...@gmail.com>
Reviewed-by: Alex Rodoni <ar...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/cd2aad45
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/cd2aad45
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/cd2aad45

Branch: refs/heads/master
Commit: cd2aad4595e533be560aafa1f591d4f533f4283f
Parents: 1c0eeaf
Author: stiga-huang <hu...@gmail.com>
Authored: Mon May 28 19:04:51 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Wed Jun 6 21:34:22 2018 +0000

----------------------------------------------------------------------
 docs/impala.ditamap                 |   1 +
 docs/shared/impala_common.xml       |   2 +-
 docs/topics/impala_file_formats.xml |  22 ++-
 docs/topics/impala_orc.xml          | 314 +++++++++++++++++++++++++++++++
 4 files changed, 336 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/impala/blob/cd2aad45/docs/impala.ditamap
----------------------------------------------------------------------
diff --git a/docs/impala.ditamap b/docs/impala.ditamap
index 99bc6ac..18dc5b8 100644
--- a/docs/impala.ditamap
+++ b/docs/impala.ditamap
@@ -283,6 +283,7 @@ under the License.
   <topicref href="topics/impala_file_formats.xml">
     <topicref href="topics/impala_txtfile.xml"/>
     <topicref href="topics/impala_parquet.xml"/>
+    <topicref href="topics/impala_orc.xml"/>
     <topicref href="topics/impala_avro.xml"/>
     <topicref href="topics/impala_rcfile.xml"/>
     <topicref href="topics/impala_seqfile.xml"/>

http://git-wip-us.apache.org/repos/asf/impala/blob/cd2aad45/docs/shared/impala_common.xml
----------------------------------------------------------------------
diff --git a/docs/shared/impala_common.xml b/docs/shared/impala_common.xml
index 2711c3a..f432b65 100644
--- a/docs/shared/impala_common.xml
+++ b/docs/shared/impala_common.xml
@@ -1313,7 +1313,7 @@ drop database temp;
 
       <p rev="2.6.0 IMPALA-3453" id="s3_block_splitting">
         In <keyword keyref="impala26_full"/> and higher, Impala queries are optimized for files stored in Amazon S3.
-        For Impala tables that use the file formats Parquet, RCFile, SequenceFile,
+        For Impala tables that use the file formats Parquet, ORC, RCFile, SequenceFile,
         Avro, and uncompressed text, the setting <codeph>fs.s3a.block.size</codeph>
         in the <filepath>core-site.xml</filepath> configuration file determines
         how Impala divides the I/O work of reading the data files. This configuration

http://git-wip-us.apache.org/repos/asf/impala/blob/cd2aad45/docs/topics/impala_file_formats.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_file_formats.xml b/docs/topics/impala_file_formats.xml
index f68e94f..7516eae 100644
--- a/docs/topics/impala_file_formats.xml
+++ b/docs/topics/impala_file_formats.xml
@@ -110,6 +110,24 @@ under the License.
               Yes: <codeph>CREATE TABLE</codeph>, <codeph>INSERT</codeph>, <codeph>LOAD DATA</codeph>, and query.
             </entry>
           </row>
+          <row id="orc_support">
+            <entry>
+              <xref href="impala_orc.xml#orc">ORC</xref>
+            </entry>
+            <entry>
+              Structured
+            </entry>
+            <entry>
+              gzip, Snappy, LZO, LZ4; currently gzip by default
+            </entry>
+            <entry>
+              Yes, in Impala 2.12.0 and higher.
+            </entry>
+            <entry>
+              No. Import data by using <codeph>LOAD DATA</codeph> on data files already in the right format, or use
+              <codeph>INSERT</codeph> in Hive followed by <codeph>REFRESH <varname>table_name</varname></codeph> in Impala.
+            </entry>
+          </row>
           <row id="txtfile_support">
             <entry>
               <xref href="impala_txtfile.xml#txtfile">Text</xref>
@@ -201,8 +219,8 @@ under the License.
     </table>
 
     <p rev="DOCS-1370">
-      Impala can only query the file formats listed in the preceding table.
-      In particular, Impala does not support the ORC file format.
+      Impala can only query the file formats listed in the preceding table. The ORC support is an experimental feature
+      since Impala-2.12. To disable it, set --enable_orc_scanner to false when starting the cluster.
     </p>
 
     <p>

http://git-wip-us.apache.org/repos/asf/impala/blob/cd2aad45/docs/topics/impala_orc.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_orc.xml b/docs/topics/impala_orc.xml
new file mode 100644
index 0000000..3bb9a6d
--- /dev/null
+++ b/docs/topics/impala_orc.xml
@@ -0,0 +1,314 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="orc">
+
+  <title>Using the ORC File Format with Impala Tables</title>
+  <titlealts audience="PDF"><navtitle>ORC Data Files</navtitle></titlealts>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Impala"/>
+      <!-- <data name="Category" value="ORC"/> -->
+      <data name="Category" value="File Formats"/>
+      <data name="Category" value="Tables"/>
+      <data name="Category" value="Developers"/>
+      <data name="Category" value="Data Analysts"/>
+    </metadata>
+  </prolog>
+
+  <conbody>
+
+    <p>
+      <indexterm audience="hidden">ORC support in Impala</indexterm>
+      Impala supports using ORC data files as an experimental feature since 2.12.
+      To disable it, set --enable_orc_scanner to false when starting the cluster.
+    </p>
+
+    <table>
+      <title>ORC Format Support in Impala</title>
+      <tgroup cols="5">
+        <colspec colname="1" colwidth="10*"/>
+        <colspec colname="2" colwidth="10*"/>
+        <colspec colname="3" colwidth="20*"/>
+        <colspec colname="4" colwidth="30*"/>
+        <colspec colname="5" colwidth="30*"/>
+        <thead>
+          <row>
+            <entry>
+              File Type
+            </entry>
+            <entry>
+              Format
+            </entry>
+            <entry>
+              Compression Codecs
+            </entry>
+            <entry>
+              Impala Can CREATE?
+            </entry>
+            <entry>
+              Impala Can INSERT?
+            </entry>
+          </row>
+        </thead>
+        <tbody>
+          <row conref="impala_file_formats.xml#file_formats/orc_support">
+            <entry/>
+          </row>
+        </tbody>
+      </tgroup>
+    </table>
+
+    <p outputclass="toc inpage"/>
+  </conbody>
+
+  <concept id="orc_create">
+
+    <title>Creating ORC Tables and Loading Data</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="ETL"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        If you do not have an existing data file to use, begin by creating one in the appropriate format.
+      </p>
+
+      <p>
+        <b>To create an ORC table:</b>
+      </p>
+
+      <p>
+        In the <codeph>impala-shell</codeph> interpreter, issue a command similar to:
+      </p>
+
+<codeblock>CREATE TABLE orc_table (<varname>column_specs</varname>) STORED AS ORC;</codeblock>
+
+      <p>
+        Because Impala can query some kinds of tables that it cannot currently write to, after creating tables of
+        certain file formats, you might use the Hive shell to load the data. See
+        <xref href="impala_file_formats.xml#file_formats"/> for details. After loading data into a table through
+        Hive or other mechanism outside of Impala, issue a <codeph>REFRESH <varname>table_name</varname></codeph>
+        statement the next time you connect to the Impala node, before querying the table, to make Impala recognize
+        the new data.
+      </p>
+
+      <p>
+        For example, here is how you might create some ORC tables in Impala (by specifying the columns
+        explicitly, or cloning the structure of another table), load data through Hive, and query them through
+        Impala:
+      </p>
+
+<codeblock>$ impala-shell -i localhost
+[localhost:21000] default&gt; CREATE TABLE orc_table (x INT) STORED AS ORC;
+[localhost:21000] default&gt; CREATE TABLE orc_clone LIKE some_other_table STORED AS ORC;
+[localhost:21000] default&gt; quit;
+
+$ hive
+hive&gt; INSERT INTO TABLE orc_table SELECT x FROM some_other_table;
+3 Rows loaded to orc_table
+Time taken: 4.169 seconds
+hive&gt; quit;
+
+$ impala-shell -i localhost
+[localhost:21000] default&gt; SELECT * FROM orc_table;
+Fetched 0 row(s) in 0.11s
+[localhost:21000] default&gt; -- Make Impala recognize the data loaded through Hive;
+[localhost:21000] default&gt; REFRESH orc_table;
+[localhost:21000] default&gt; SELECT * FROM orc_table;
++---+
+| x |
++---+
+| 1 |
+| 2 |
+| 3 |
++---+
+Fetched 3 row(s) in 0.11s</codeblock>
+
+    </conbody>
+  </concept>
+
+  <concept id="orc_compression">
+
+    <title>Enabling Compression for ORC Tables</title>
+  <prolog>
+    <metadata>
+      <data name="Category" value="Snappy"/>
+      <data name="Category" value="Compression"/>
+    </metadata>
+  </prolog>
+
+    <conbody>
+
+      <p>
+        <indexterm audience="hidden">compression</indexterm>
+        ORC tables are in zlib (Deflate in Impala) compression in default. You may want
+        to use Snappy or LZO compression on existing tables for different balance between
+        compression ratio and decompression speed. In Hive-1.1.0, the supported
+        compressions for ORC tables are NONE, ZLIB, SNAPPY and LZO.
+        For example, to enable Snappy compression, you would specify
+        the following additional settings when loading data through the Hive shell:
+      </p>
+
+<codeblock>hive&gt; SET hive.exec.compress.output=true;
+hive&gt; SET orc.compress=SNAPPY;
+hive&gt; INSERT OVERWRITE TABLE <varname>new_table</varname> SELECT * FROM <varname>old_table</varname>;</codeblock>
+
+      <p>
+        If you are converting partitioned tables, you must complete additional steps. In such a case, specify
+        additional settings similar to the following:
+      </p>
+
+<codeblock>hive&gt; CREATE TABLE <varname>new_table</varname> (<varname>your_cols</varname>) PARTITIONED BY (<varname>partition_cols</varname>) STORED AS <varname>new_format</varname>;
+hive&gt; SET hive.exec.dynamic.partition.mode=nonstrict;
+hive&gt; SET hive.exec.dynamic.partition=true;
+hive&gt; INSERT OVERWRITE TABLE <varname>new_table</varname> PARTITION(<varname>comma_separated_partition_cols</varname>) SELECT * FROM <varname>old_table</varname>;</codeblock>
+
+      <p>
+        Remember that Hive does not require that you specify a source format for it. Consider the case of
+        converting a table with two partition columns called <codeph>year</codeph> and <codeph>month</codeph> to a
+        Snappy compressed ORC table. Combining the components outlined previously to complete this table conversion,
+        you would specify settings similar to the following:
+      </p>
+
+<codeblock>hive&gt; CREATE TABLE tbl_orc (int_col INT, string_col STRING) STORED AS ORC;
+hive&gt; SET hive.exec.compress.output=true;
+hive&gt; SET orc.compress=SNAPPY;
+hive&gt; SET hive.exec.dynamic.partition.mode=nonstrict;
+hive&gt; SET hive.exec.dynamic.partition=true;
+hive&gt; INSERT OVERWRITE TABLE tbl_orc SELECT * FROM tbl;</codeblock>
+
+      <p>
+        To complete a similar process for a table that includes partitions, you would specify settings similar to
+        the following:
+      </p>
+
+<codeblock>hive&gt; CREATE TABLE tbl_orc (int_col INT, string_col STRING) PARTITIONED BY (year INT) STORED AS ORC;
+hive&gt; SET hive.exec.compress.output=true;
+hive&gt; SET orc.compress=SNAPPY;
+hive&gt; SET hive.exec.dynamic.partition.mode=nonstrict;
+hive&gt; SET hive.exec.dynamic.partition=true;
+hive&gt; INSERT OVERWRITE TABLE tbl_orc PARTITION(year) SELECT * FROM tbl;</codeblock>
+
+      <note>
+        <p>
+          The compression type is specified in the following command:
+        </p>
+<codeblock>SET orc.compress=SNAPPY;</codeblock>
+        <p>
+          You could elect to specify alternative codecs such as <codeph>NONE, GZIP, LZO</codeph> here.
+        </p>
+      </note>
+    </conbody>
+  </concept>
+
+  <concept id="rcfile_performance">
+
+    <title>Query Performance for Impala ORC Tables</title>
+
+    <conbody>
+
+      <p>
+        In general, expect query performance with ORC tables to be
+        faster than with tables using text data, but slower than with
+        Parquet tables since there're bunch of optimizations for Parquet.
+        See <xref href="impala_parquet.xml#parquet"/>
+        for information about using the Parquet file format for
+        high-performance analytic queries.
+      </p>
+
+      <p conref="../shared/impala_common.xml#common/s3_block_splitting"/>
+
+    </conbody>
+  </concept>
+
+  <concept id="orc_data_types">
+
+    <title>Data Type Considerations for ORC Tables</title>
+
+    <conbody>
+
+      <p>
+        The ORC format defines a set of data types whose names differ from the names of the corresponding
+        Impala data types. If you are preparing ORC files using other Hadoop components such as Pig or
+        MapReduce, you might need to work with the type names defined by ORC. The following figure lists the
+        ORC-defined types and the equivalent types in Impala.
+      </p>
+
+      <p>
+        <b>Primitive types:</b>
+      </p>
+
+<codeblock>BINARY -&gt; STRING
+BOOLEAN -&gt; BOOLEAN
+DOUBLE -&gt; DOUBLE
+FLOAT -&gt; FLOAT
+TINYINT -&gt; TINYINT
+SMALLINT -&gt; SMALLINT
+INT -&gt; INT
+BIGINT -&gt; BIGINT
+TIMESTAMP -&gt; TIMESTAMP
+DATE (not supported)
+</codeblock>
+
+      <p>
+        <b>Complex types:</b>
+      </p>
+      <p>
+        Complex types are currently not supported on ORC. However, queries materializing only scalar type
+        columns are allowed:
+      </p>
+
+<codeblock>$ hive
+hive&gt; CREATE TABLE orc_nested_table (id INT, a ARRAY&lt;INT&gt;) STORED AS ORC;
+hive&gt; INSERT INTO TABLE orc_nested_table SELECT 1, ARRAY(1,2,3);
+OK
+Time taken: 2.629 seconds
+hive&gt; quit;
+
+$ impala-shell -i localhost
+[localhost:21000] default&gt; INVALIDATE METADATA orc_nested_table;
+[localhost:21000] default&gt; SELECT 1 FROM orc_nested_table t, t.a;
+ERROR: NotImplementedException: Scan of table 't' in format 'ORC' is not supported because the table has a column 'a' with a complex type 'ARRAY&lt;INT&gt;'.
+Complex types are supported for these file formats: PARQUET.
+
+[localhost:21000] default&gt; SELECT COUNT(*) FROM orc_nested_table;
++----------+
+| count(*) |
++----------+
+| 1        |
++----------+
+Fetched 1 row(s) in 0.12s
+
+[localhost:21000] default&gt; SELECT id FROM orc_nested_table;
++----+
+| id |
++----+
+| 1  |
++----+
+Fetched 1 row(s) in 0.12s
+</codeblock>
+
+    </conbody>
+  </concept>
+</concept>