You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2021/01/09 01:02:19 UTC
[impala] 01/04: IMPALA-10091: [DOCS] add
REFRESH_UPDATED_HMS_PARTITIONS query option
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 44bade8e7f14d22b6a16650d54dc9bee62caeb50
Author: Shajini Thayasingh <st...@cloudera.com>
AuthorDate: Tue Jan 5 12:49:21 2021 -0800
IMPALA-10091: [DOCS] add REFRESH_UPDATED_HMS_PARTITIONS query option
remove trailing spaces
added this new query option for Impala 4.0
Change-Id: I95b31b33f99073c57752e66eaf0f34facf511fc6
Reviewed-on: http://gerrit.cloudera.org:8080/16925
Reviewed-by: Vihang Karajgaonkar <vi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
docs/impala.ditamap | 1 +
docs/topics/impala_refresh_updated_hms.xml | 72 ++++++++++++++++++++++++++++++
2 files changed, 73 insertions(+)
diff --git a/docs/impala.ditamap b/docs/impala.ditamap
index d6aeedd..cf7e3b6 100644
--- a/docs/impala.ditamap
+++ b/docs/impala.ditamap
@@ -230,6 +230,7 @@ under the License.
<topicref href="topics/impala_parquet_write_page_index.xml"/>
<topicref rev="2.6.0 IMPALA-3286" href="topics/impala_prefetch_mode.xml"/>
<topicref href="topics/impala_query_timeout_s.xml"/>
+ <topicref href="topics/impala_refresh_updated_hms.xml"/>
<topicref rev="2.7.0" href="topics/impala_replica_preference.xml"/>
<topicref href="topics/impala_request_pool.xml"/>
<topicref href="topics/impala_resource_trace_ratio.xml"/>
diff --git a/docs/topics/impala_refresh_updated_hms.xml b/docs/topics/impala_refresh_updated_hms.xml
new file mode 100644
index 0000000..779c7d2
--- /dev/null
+++ b/docs/topics/impala_refresh_updated_hms.xml
@@ -0,0 +1,72 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="4.0 IMPALA-10091" id="refresh_updated_hms">
+
+ <title>REFRESH_UPDATED_HMS_PARTITIONS Query Option</title>
+
+ <titlealts audience="PDF">
+
+ <navtitle>REFRESH_UPDATED_HMS_PARTITIONS</navtitle>
+
+ </titlealts>
+
+ <prolog>
+ <metadata>
+ <data name="Category" value="Impala"/>
+ <data name="Category" value="Impala Query Options"/>
+ <data name="Category" value="Querying"/>
+ <data name="Category" value="Developers"/>
+ <data name="Category" value="Administrators"/>
+ </metadata>
+ </prolog>
+
+ <conbody>
+
+ <p>
+ As the name implies the query option <codeph>REFRESH_UPDATED_HMS_PARTITIONS</codeph> is used
+ to refresh any updated HMS partitions.
+ </p>
+
+ <p>This option is disabled by default so that the performance is not compromised when refreshing a
+ table. However, for certain corner case scenarios refresh table command does not detect changed
+ partitions. In case of the default refresh, catalogd detects, adds any new partitions and removes
+ any partitions which are not present in HMS anymore. However, it does not update any partitions
+ that changed (eg. change of location). When this query option is enabled, the refresh table
+ command will detect certain changes to the partitions and update them accordingly. Currently,
+ catalogd will update the partitions if any of the following StorageDescriptor properties have
+ been modified. 1. Partition Location 2. Partition Fileformat. 3 SerdeInfo. 4. Partition schema
+ changes. 5. Partition bucketInfo.</p>
+
+ <p>
+ <b>Type:</b><codeph>BOOLEAN</codeph>; recognized values are 1 and 0, or true and false; any other
+ value will be interpreted as false.
+ </p>
+
+ <p>
+ <b>Default:</b><codeph>FALSE</codeph> (shown as 0 in output of SET statement).
+ </p>
+
+ <p><b>Added in:</b>
+ <keyword keyref="impala40"/></p>
+
+ </conbody>
+
+</concept>