You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ar...@apache.org on 2019/10/03 23:03:55 UTC
[impala] 03/03: IMPALA-8971: [DOCS] Documented the new query option
for Parquet split size
This is an automated email from the ASF dual-hosted git repository.
arodoni pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 38290f91311bec57d2658d046ee27ed6901329ea
Author: Alex Rodoni <ar...@cloudera.com>
AuthorDate: Tue Oct 1 14:57:06 2019 -0700
IMPALA-8971: [DOCS] Documented the new query option for Parquet split size
Change-Id: I6343eec91df7ae01c342b5800d08248c1c4a739a
Reviewed-on: http://gerrit.cloudera.org:8080/14338
Reviewed-by: Sahil Takiar <st...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
docs/impala.ditamap | 1 +
docs/topics/impala_parquet.xml | 4 ++
.../impala_parquet_object_store_split_size.xml | 48 +++++++++++++++++
docs/topics/impala_s3.xml | 60 ++++++----------------
4 files changed, 68 insertions(+), 45 deletions(-)
diff --git a/docs/impala.ditamap b/docs/impala.ditamap
index 33224a6..839e027 100644
--- a/docs/impala.ditamap
+++ b/docs/impala.ditamap
@@ -218,6 +218,7 @@ under the License.
<topicref href="topics/impala_parquet_dictionary_filtering.xml"/>
<topicref rev="2.6.0 IMPALA-2835" href="topics/impala_parquet_fallback_schema_resolution.xml"/>
<topicref href="topics/impala_parquet_file_size.xml"/>
+ <topicref href="topics/impala_parquet_object_store_split_size.xml"/>
<topicref href="topics/impala_parquet_page_row_count_limit.xml"/>
<topicref href="topics/impala_parquet_read_statistics.xml"/>
<topicref href="topics/impala_parquet_read_page_index.xml"/>
diff --git a/docs/topics/impala_parquet.xml b/docs/topics/impala_parquet.xml
index 8148973..01cdb26 100644
--- a/docs/topics/impala_parquet.xml
+++ b/docs/topics/impala_parquet.xml
@@ -336,6 +336,10 @@ under the License.
</p>
<p conref="../shared/impala_common.xml#common/s3_block_splitting"/>
+ <p>Starting in Impala 3.4.0, use the query option
+ <codeph>PARQUET_OBJECT_STORE_SPLIT_SIZE</codeph> to control the
+ Parquet split size for non-block stores (e.g. S3, ADLS, etc.). The
+ default value is 256 MB.</p>
<p rev="IMPALA-3909">
In <keyword keyref="impala29"/> and higher, Parquet files written by Impala include
diff --git a/docs/topics/impala_parquet_object_store_split_size.xml b/docs/topics/impala_parquet_object_store_split_size.xml
new file mode 100644
index 0000000..4b315c5
--- /dev/null
+++ b/docs/topics/impala_parquet_object_store_split_size.xml
@@ -0,0 +1,48 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="parquet_block_size" id="parquet_object_store_split_size">
+ <title>PARQUET_OBJECT_STORE_SPLIT_SIZE Query Option</title>
+ <titlealts audience="PDF">
+ <navtitle>PARQUET_OBJECT_STORE_SPLIT_SIZE</navtitle>
+ </titlealts>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Impala"/>
+ <data name="Category" value="Parquet"/>
+ <data name="Category" value="ETL"/>
+ <data name="Category" value="File Formats"/>
+ <data name="Category" value="Impala Query Options"/>
+ <data name="Category" value="Developers"/>
+ <data name="Category" value="Data Analysts"/>
+ </metadata>
+ </prolog>
+ <conbody>
+ <p>Use the <codeph>PARQUET_OBJECT_STORE_SPLIT_SIZE</codeph> query option to
+ control the Parquet format specific split sizes on non-block based stores,
+ e.g. S3, ADLS, etc.</p>
+ <p>The value must be greater than or equal to 1 MB.</p>
+ <p><b>Type:</b>
+ <codeph>INT</codeph></p>
+ <p><b>Default:</b> 256 MB</p>
+ <p>Usage notes:</p>
+ <p><b>Added in:</b> Impala 3.4</p>
+ </conbody>
+</concept>
diff --git a/docs/topics/impala_s3.xml b/docs/topics/impala_s3.xml
index e67d9db..31fc348 100644
--- a/docs/topics/impala_s3.xml
+++ b/docs/topics/impala_s3.xml
@@ -38,15 +38,12 @@ under the License.
<note conref="../shared/impala_common.xml#common/s3_production"/>
- <p rev="2.2.0">
- <indexterm audience="hidden">S3 with Impala</indexterm>
-
- <indexterm audience="hidden">Amazon S3 with Impala</indexterm>
- You can use Impala to query data residing on the Amazon S3 filesystem. This capability allows convenient
- access to a storage system that is remotely managed, accessible from anywhere, and integrated with various
- cloud-based services. Impala can query files in any supported file format from S3. The S3 storage location
- can be for an entire table, or individual partitions in a partitioned table.
- </p>
+ <p rev="2.2.0"> You can use Impala to query data residing on the Amazon S3
+ filesystem. This capability allows convenient access to a storage system
+ that is remotely managed, accessible from anywhere, and integrated with
+ various cloud-based services. Impala can query files in any supported file
+ format from S3. The S3 storage location can be for an entire table, or
+ individual partitions in a partitioned table. </p>
<p>
The default Impala tables use data files stored on HDFS, which are ideal for bulk loads and queries using
@@ -133,9 +130,6 @@ under the License.
To allow Impala to access data in S3, specify values for the following configuration settings in your
<filepath>core-site.xml</filepath> file:
</p>
-
-<!-- Normally I would turn this example into CDATA notation to avoid all the < and > entities.
- However, then I couldn't use the <varname> tag inside the same example. -->
<codeblock>
<property>
<name>fs.s3a.access.key</name>
@@ -154,18 +148,6 @@ under the License.
</p>
<note type="important">
-<!--
- <ul>
- <li>
- <p rev="IMPALA-3306">
- The <codeph>s3a_access_key_cmd</codeph> and <codeph>s3a_secret_key_cmd</codeph> settings
- for <cmdname>impalad</cmdname> only allow Impala to access S3. You must still include the credentials in the
- client <filepath>hdfs-site.xml</filepath> configuration file to allow S3 access for the Hive metastore,
- <codeph>hadoop fs</codeph> command, and so on.
- </p>
- </li>
- <li>
--->
<p>
Although you can specify the access key ID and secret key as part of the <codeph>s3a://</codeph> URL in the
<codeph>LOCATION</codeph> attribute, doing so makes this sensitive information visible in many places, such
@@ -173,23 +155,6 @@ under the License.
centrally in the <filepath>core-site.xml</filepath> file, and restrict read access to that file to only
trusted users.
</p>
-<!--
- </li>
--->
- <!-- Overriding with a new first list bullet following clarification by Sailesh.
- <li>
- <p rev="IMPALA-3306">
- Prior to <keyword keyref="impala26_full"/> an alternative way to specify the keys was by
- including the fields <codeph>fs.s3a.access.key</codeph> and <codeph>fs.s3a.secret.key</codeph>
- in a configuration file such as <filepath>core-site.xml</filepath> or <filepath>hdfs-site.xml</filepath>.
- With the enhanced S3 key management in <keyword keyref="impala26_full"/> and higher, if you are upgrading from
- an earlier release where you used Impala with S3, remove the S3 keys from any copies of those files.
- </p>
- </li>
- -->
-<!--
- </ul>
--->
</note>
</conbody>
@@ -643,6 +608,10 @@ under the License.
</p>
<p conref="../shared/impala_common.xml#common/s3_block_splitting"/>
+ <p>Starting in Impala 3.4.0, use the query option
+ <codeph>PARQUET_OBJECT_STORE_SPLIT_SIZE</codeph> to control the
+ Parquet split size for non-block stores (e.g. S3, ADLS, etc.). The
+ default value is 256 MB.</p>
<p conref="../shared/impala_common.xml#common/s3_dml_performance"/>
@@ -733,10 +702,11 @@ under the License.
<p> Set <codeph>fs.s3a.block.size</codeph> to 134217728 (128 MB in
bytes) if most Parquet files queried by Impala were written by Hive
or ParquetMR jobs. Set the block size to 268435456 (256 MB in bytes)
- if most Parquet files queried by Impala were written by Impala.
- As of Impala 3.4.0, the query option <codeph>PARQUET_OBJECT_STORE_SPLIT_SIZE</codeph>
- controls the Parquet split size for non-block stores (e.g. S3, ADLS, etc.).
- The default value is 256 MB.</p>
+ if most Parquet files queried by Impala were written by Impala. </p>
+ <p>As of Impala 3.4.0, the query option
+ <codeph>PARQUET_OBJECT_STORE_SPLIT_SIZE</codeph> controls the
+ Parquet split size for non-block stores (e.g. S3, ADLS, etc.). The
+ default value is 256 MB.</p>
</li>
<li>
<p>