You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2018/06/07 00:24:20 UTC
[4/4] impala git commit: IMPALA-6714: [DOCS] ORC file format support
IMPALA-6714: [DOCS] ORC file format support
This document is wrote refering to RCFile and Parquet's docs.
The orc-support patch was merged in impala-2.12 and impala-3.0,
so we start to support ORC format as an experimental feature
since impala-2.12.
Change-Id: Ib1ee23ed844653c274babdce5a332dbe5c79b630
Reviewed-on: http://gerrit.cloudera.org:8080/10525
Reviewed-by: Balazs Jeszenszky <je...@gmail.com>
Reviewed-by: Alex Rodoni <ar...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/cd2aad45
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/cd2aad45
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/cd2aad45
Branch: refs/heads/master
Commit: cd2aad4595e533be560aafa1f591d4f533f4283f
Parents: 1c0eeaf
Author: stiga-huang <hu...@gmail.com>
Authored: Mon May 28 19:04:51 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Wed Jun 6 21:34:22 2018 +0000
----------------------------------------------------------------------
docs/impala.ditamap | 1 +
docs/shared/impala_common.xml | 2 +-
docs/topics/impala_file_formats.xml | 22 ++-
docs/topics/impala_orc.xml | 314 +++++++++++++++++++++++++++++++
4 files changed, 336 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/cd2aad45/docs/impala.ditamap
----------------------------------------------------------------------
diff --git a/docs/impala.ditamap b/docs/impala.ditamap
index 99bc6ac..18dc5b8 100644
--- a/docs/impala.ditamap
+++ b/docs/impala.ditamap
@@ -283,6 +283,7 @@ under the License.
<topicref href="topics/impala_file_formats.xml">
<topicref href="topics/impala_txtfile.xml"/>
<topicref href="topics/impala_parquet.xml"/>
+ <topicref href="topics/impala_orc.xml"/>
<topicref href="topics/impala_avro.xml"/>
<topicref href="topics/impala_rcfile.xml"/>
<topicref href="topics/impala_seqfile.xml"/>
http://git-wip-us.apache.org/repos/asf/impala/blob/cd2aad45/docs/shared/impala_common.xml
----------------------------------------------------------------------
diff --git a/docs/shared/impala_common.xml b/docs/shared/impala_common.xml
index 2711c3a..f432b65 100644
--- a/docs/shared/impala_common.xml
+++ b/docs/shared/impala_common.xml
@@ -1313,7 +1313,7 @@ drop database temp;
<p rev="2.6.0 IMPALA-3453" id="s3_block_splitting">
In <keyword keyref="impala26_full"/> and higher, Impala queries are optimized for files stored in Amazon S3.
- For Impala tables that use the file formats Parquet, RCFile, SequenceFile,
+ For Impala tables that use the file formats Parquet, ORC, RCFile, SequenceFile,
Avro, and uncompressed text, the setting <codeph>fs.s3a.block.size</codeph>
in the <filepath>core-site.xml</filepath> configuration file determines
how Impala divides the I/O work of reading the data files. This configuration
http://git-wip-us.apache.org/repos/asf/impala/blob/cd2aad45/docs/topics/impala_file_formats.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_file_formats.xml b/docs/topics/impala_file_formats.xml
index f68e94f..7516eae 100644
--- a/docs/topics/impala_file_formats.xml
+++ b/docs/topics/impala_file_formats.xml
@@ -110,6 +110,24 @@ under the License.
Yes: <codeph>CREATE TABLE</codeph>, <codeph>INSERT</codeph>, <codeph>LOAD DATA</codeph>, and query.
</entry>
</row>
+ <row id="orc_support">
+ <entry>
+ <xref href="impala_orc.xml#orc">ORC</xref>
+ </entry>
+ <entry>
+ Structured
+ </entry>
+ <entry>
+ gzip, Snappy, LZO, LZ4; currently gzip by default
+ </entry>
+ <entry>
+ Yes, in Impala 2.12.0 and higher.
+ </entry>
+ <entry>
+ No. Import data by using <codeph>LOAD DATA</codeph> on data files already in the right format, or use
+ <codeph>INSERT</codeph> in Hive followed by <codeph>REFRESH <varname>table_name</varname></codeph> in Impala.
+ </entry>
+ </row>
<row id="txtfile_support">
<entry>
<xref href="impala_txtfile.xml#txtfile">Text</xref>
@@ -201,8 +219,8 @@ under the License.
</table>
<p rev="DOCS-1370">
- Impala can only query the file formats listed in the preceding table.
- In particular, Impala does not support the ORC file format.
+ Impala can only query the file formats listed in the preceding table. The ORC support is an experimental feature
+ since Impala-2.12. To disable it, set --enable_orc_scanner to false when starting the cluster.
</p>
<p>
http://git-wip-us.apache.org/repos/asf/impala/blob/cd2aad45/docs/topics/impala_orc.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_orc.xml b/docs/topics/impala_orc.xml
new file mode 100644
index 0000000..3bb9a6d
--- /dev/null
+++ b/docs/topics/impala_orc.xml
@@ -0,0 +1,314 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept id="orc">
+
+ <title>Using the ORC File Format with Impala Tables</title>
+ <titlealts audience="PDF"><navtitle>ORC Data Files</navtitle></titlealts>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Impala"/>
+ <!-- <data name="Category" value="ORC"/> -->
+ <data name="Category" value="File Formats"/>
+ <data name="Category" value="Tables"/>
+ <data name="Category" value="Developers"/>
+ <data name="Category" value="Data Analysts"/>
+ </metadata>
+ </prolog>
+
+ <conbody>
+
+ <p>
+ <indexterm audience="hidden">ORC support in Impala</indexterm>
+ Impala supports using ORC data files as an experimental feature since 2.12.
+ To disable it, set --enable_orc_scanner to false when starting the cluster.
+ </p>
+
+ <table>
+ <title>ORC Format Support in Impala</title>
+ <tgroup cols="5">
+ <colspec colname="1" colwidth="10*"/>
+ <colspec colname="2" colwidth="10*"/>
+ <colspec colname="3" colwidth="20*"/>
+ <colspec colname="4" colwidth="30*"/>
+ <colspec colname="5" colwidth="30*"/>
+ <thead>
+ <row>
+ <entry>
+ File Type
+ </entry>
+ <entry>
+ Format
+ </entry>
+ <entry>
+ Compression Codecs
+ </entry>
+ <entry>
+ Impala Can CREATE?
+ </entry>
+ <entry>
+ Impala Can INSERT?
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row conref="impala_file_formats.xml#file_formats/orc_support">
+ <entry/>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <p outputclass="toc inpage"/>
+ </conbody>
+
+ <concept id="orc_create">
+
+ <title>Creating ORC Tables and Loading Data</title>
+ <prolog>
+ <metadata>
+ <data name="Category" value="ETL"/>
+ </metadata>
+ </prolog>
+
+ <conbody>
+
+ <p>
+ If you do not have an existing data file to use, begin by creating one in the appropriate format.
+ </p>
+
+ <p>
+ <b>To create an ORC table:</b>
+ </p>
+
+ <p>
+ In the <codeph>impala-shell</codeph> interpreter, issue a command similar to:
+ </p>
+
+<codeblock>CREATE TABLE orc_table (<varname>column_specs</varname>) STORED AS ORC;</codeblock>
+
+ <p>
+ Because Impala can query some kinds of tables that it cannot currently write to, after creating tables of
+ certain file formats, you might use the Hive shell to load the data. See
+ <xref href="impala_file_formats.xml#file_formats"/> for details. After loading data into a table through
+ Hive or other mechanism outside of Impala, issue a <codeph>REFRESH <varname>table_name</varname></codeph>
+ statement the next time you connect to the Impala node, before querying the table, to make Impala recognize
+ the new data.
+ </p>
+
+ <p>
+ For example, here is how you might create some ORC tables in Impala (by specifying the columns
+ explicitly, or cloning the structure of another table), load data through Hive, and query them through
+ Impala:
+ </p>
+
+<codeblock>$ impala-shell -i localhost
+[localhost:21000] default> CREATE TABLE orc_table (x INT) STORED AS ORC;
+[localhost:21000] default> CREATE TABLE orc_clone LIKE some_other_table STORED AS ORC;
+[localhost:21000] default> quit;
+
+$ hive
+hive> INSERT INTO TABLE orc_table SELECT x FROM some_other_table;
+3 Rows loaded to orc_table
+Time taken: 4.169 seconds
+hive> quit;
+
+$ impala-shell -i localhost
+[localhost:21000] default> SELECT * FROM orc_table;
+Fetched 0 row(s) in 0.11s
+[localhost:21000] default> -- Make Impala recognize the data loaded through Hive;
+[localhost:21000] default> REFRESH orc_table;
+[localhost:21000] default> SELECT * FROM orc_table;
++---+
+| x |
++---+
+| 1 |
+| 2 |
+| 3 |
++---+
+Fetched 3 row(s) in 0.11s</codeblock>
+
+ </conbody>
+ </concept>
+
+ <concept id="orc_compression">
+
+ <title>Enabling Compression for ORC Tables</title>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Snappy"/>
+ <data name="Category" value="Compression"/>
+ </metadata>
+ </prolog>
+
+ <conbody>
+
+ <p>
+ <indexterm audience="hidden">compression</indexterm>
+ ORC tables are in zlib (Deflate in Impala) compression in default. You may want
+ to use Snappy or LZO compression on existing tables for different balance between
+ compression ratio and decompression speed. In Hive-1.1.0, the supported
+ compressions for ORC tables are NONE, ZLIB, SNAPPY and LZO.
+ For example, to enable Snappy compression, you would specify
+ the following additional settings when loading data through the Hive shell:
+ </p>
+
+<codeblock>hive> SET hive.exec.compress.output=true;
+hive> SET orc.compress=SNAPPY;
+hive> INSERT OVERWRITE TABLE <varname>new_table</varname> SELECT * FROM <varname>old_table</varname>;</codeblock>
+
+ <p>
+ If you are converting partitioned tables, you must complete additional steps. In such a case, specify
+ additional settings similar to the following:
+ </p>
+
+<codeblock>hive> CREATE TABLE <varname>new_table</varname> (<varname>your_cols</varname>) PARTITIONED BY (<varname>partition_cols</varname>) STORED AS <varname>new_format</varname>;
+hive> SET hive.exec.dynamic.partition.mode=nonstrict;
+hive> SET hive.exec.dynamic.partition=true;
+hive> INSERT OVERWRITE TABLE <varname>new_table</varname> PARTITION(<varname>comma_separated_partition_cols</varname>) SELECT * FROM <varname>old_table</varname>;</codeblock>
+
+ <p>
+ Remember that Hive does not require that you specify a source format for it. Consider the case of
+ converting a table with two partition columns called <codeph>year</codeph> and <codeph>month</codeph> to a
+ Snappy compressed ORC table. Combining the components outlined previously to complete this table conversion,
+ you would specify settings similar to the following:
+ </p>
+
+<codeblock>hive> CREATE TABLE tbl_orc (int_col INT, string_col STRING) STORED AS ORC;
+hive> SET hive.exec.compress.output=true;
+hive> SET orc.compress=SNAPPY;
+hive> SET hive.exec.dynamic.partition.mode=nonstrict;
+hive> SET hive.exec.dynamic.partition=true;
+hive> INSERT OVERWRITE TABLE tbl_orc SELECT * FROM tbl;</codeblock>
+
+ <p>
+ To complete a similar process for a table that includes partitions, you would specify settings similar to
+ the following:
+ </p>
+
+<codeblock>hive> CREATE TABLE tbl_orc (int_col INT, string_col STRING) PARTITIONED BY (year INT) STORED AS ORC;
+hive> SET hive.exec.compress.output=true;
+hive> SET orc.compress=SNAPPY;
+hive> SET hive.exec.dynamic.partition.mode=nonstrict;
+hive> SET hive.exec.dynamic.partition=true;
+hive> INSERT OVERWRITE TABLE tbl_orc PARTITION(year) SELECT * FROM tbl;</codeblock>
+
+ <note>
+ <p>
+ The compression type is specified in the following command:
+ </p>
+<codeblock>SET orc.compress=SNAPPY;</codeblock>
+ <p>
+ You could elect to specify alternative codecs such as <codeph>NONE, GZIP, LZO</codeph> here.
+ </p>
+ </note>
+ </conbody>
+ </concept>
+
+ <concept id="rcfile_performance">
+
+ <title>Query Performance for Impala ORC Tables</title>
+
+ <conbody>
+
+ <p>
+ In general, expect query performance with ORC tables to be
+ faster than with tables using text data, but slower than with
+ Parquet tables since there're bunch of optimizations for Parquet.
+ See <xref href="impala_parquet.xml#parquet"/>
+ for information about using the Parquet file format for
+ high-performance analytic queries.
+ </p>
+
+ <p conref="../shared/impala_common.xml#common/s3_block_splitting"/>
+
+ </conbody>
+ </concept>
+
+ <concept id="orc_data_types">
+
+ <title>Data Type Considerations for ORC Tables</title>
+
+ <conbody>
+
+ <p>
+ The ORC format defines a set of data types whose names differ from the names of the corresponding
+ Impala data types. If you are preparing ORC files using other Hadoop components such as Pig or
+ MapReduce, you might need to work with the type names defined by ORC. The following figure lists the
+ ORC-defined types and the equivalent types in Impala.
+ </p>
+
+ <p>
+ <b>Primitive types:</b>
+ </p>
+
+<codeblock>BINARY -> STRING
+BOOLEAN -> BOOLEAN
+DOUBLE -> DOUBLE
+FLOAT -> FLOAT
+TINYINT -> TINYINT
+SMALLINT -> SMALLINT
+INT -> INT
+BIGINT -> BIGINT
+TIMESTAMP -> TIMESTAMP
+DATE (not supported)
+</codeblock>
+
+ <p>
+ <b>Complex types:</b>
+ </p>
+ <p>
+ Complex types are currently not supported on ORC. However, queries materializing only scalar type
+ columns are allowed:
+ </p>
+
+<codeblock>$ hive
+hive> CREATE TABLE orc_nested_table (id INT, a ARRAY<INT>) STORED AS ORC;
+hive> INSERT INTO TABLE orc_nested_table SELECT 1, ARRAY(1,2,3);
+OK
+Time taken: 2.629 seconds
+hive> quit;
+
+$ impala-shell -i localhost
+[localhost:21000] default> INVALIDATE METADATA orc_nested_table;
+[localhost:21000] default> SELECT 1 FROM orc_nested_table t, t.a;
+ERROR: NotImplementedException: Scan of table 't' in format 'ORC' is not supported because the table has a column 'a' with a complex type 'ARRAY<INT>'.
+Complex types are supported for these file formats: PARQUET.
+
+[localhost:21000] default> SELECT COUNT(*) FROM orc_nested_table;
++----------+
+| count(*) |
++----------+
+| 1 |
++----------+
+Fetched 1 row(s) in 0.12s
+
+[localhost:21000] default> SELECT id FROM orc_nested_table;
++----+
+| id |
++----+
+| 1 |
++----+
+Fetched 1 row(s) in 0.12s
+</codeblock>
+
+ </conbody>
+ </concept>
+</concept>