You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ar...@apache.org on 2018/10/31 18:34:39 UTC
[4/4] impala git commit: IMPALA-7614: [DOCS] Document the New
Invalidate Options
IMPALA-7614: [DOCS] Document the New Invalidate Options
--invalidate_tables_timeout_s
--invalidate_tables_on_memory_pressure
Change-Id: I40c552eeaee81ee6528d9f725bd416b51d8ab837
Reviewed-on: http://gerrit.cloudera.org:8080/11809
Reviewed-by: Tianyi Wang <tw...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/01f60d93
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/01f60d93
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/01f60d93
Branch: refs/heads/master
Commit: 01f60d9389a52453c346d11e36a6bce6ed0d2fcd
Parents: f7794cf
Author: Alex Rodoni <ar...@cloudera.com>
Authored: Thu Oct 25 19:02:45 2018 -0700
Committer: Alex Rodoni <ar...@cloudera.com>
Committed: Wed Oct 31 01:06:23 2018 +0000
----------------------------------------------------------------------
docs/topics/impala_config_options.xml | 69 +++++++++++++++---------------
1 file changed, 35 insertions(+), 34 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/01f60d93/docs/topics/impala_config_options.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_config_options.xml b/docs/topics/impala_config_options.xml
index 2f8a83c..7dc1add 100644
--- a/docs/topics/impala_config_options.xml
+++ b/docs/topics/impala_config_options.xml
@@ -262,30 +262,6 @@ Starting Impala Catalog Server: [ OK ]</codeblock>
</conbody>
- <concept audience="hidden" id="config_options_statestored_details">
-
- <title>Configuration Options for statestored Daemon</title>
-
- <conbody>
-
- <p></p>
-
- </conbody>
-
- </concept>
-
- <concept audience="hidden" id="config_options_catalogd_details">
-
- <title>Configuration Options for catalogd Daemon</title>
-
- <conbody>
-
- <p></p>
-
- </conbody>
-
- </concept>
-
</concept>
<concept id="config_options_checking">
@@ -348,11 +324,10 @@ Starting Impala Catalog Server: [ OK ]</codeblock>
<conbody>
- <p>
- The <cmdname>statestored</cmdname> daemon implements the Impala statestore service,
- which monitors the availability of Impala services across the cluster, and handles
- situations such as nodes becoming unavailable or becoming available again.
- </p>
+ <p> The <cmdname>statestored</cmdname> daemon implements the Impala
+ StateStore service, which monitors the availability of Impala services
+ across the cluster, and handles situations such as nodes becoming
+ unavailable or becoming available again. </p>
</conbody>
@@ -364,16 +339,42 @@ Starting Impala Catalog Server: [ OK ]</codeblock>
<conbody>
- <p>
- The <cmdname>catalogd</cmdname> daemon implements the Impala catalog service, which
- broadcasts metadata changes to all the Impala nodes when Impala creates a table, inserts
- data, or performs other kinds of DDL and DML operations.
- </p>
+ <p> The <cmdname>catalogd</cmdname> daemon implements the Impala Catalog
+ service, which broadcasts metadata changes to all the Impala nodes when
+ Impala creates a table, inserts data, or performs other kinds of DDL and
+ DML operations. </p>
<p conref="../shared/impala_common.xml#common/load_catalog_in_background"/>
</conbody>
</concept>
+ <concept id="auto_invalidate_metadata">
+ <title>Startup Options for Automatic Invalidation of Metadata</title>
+ <conbody>
+ <p>To keep the size of metadata small, <codeph>catalogd</codeph>
+ periodically scans all the tables and invalidates those not recently
+ used. There are two types of configurations in
+ <codeph>catalogd</codeph>.</p>
+ <ul>
+ <li>Time-based invalidation with the
+ <codeph>--invalidate_tables_timeout_s</codeph> flag:
+ <codeph>Catalogd</codeph> invalidates tables that are not recently
+ used in the specified time period. This flag needs to be applied to
+ both <codeph>impalad</codeph> and <codeph>catalogd</codeph>.</li>
+ <li>Java garbage collection-based invalidation with the
+ <codeph>--invalidate_tables_on_memory_pressure</codeph> flag: When
+ the memory pressure is high after a Java garbage collection in
+ <codeph>catalogd</codeph>, Impala invalidates a certain fraction of
+ the least recently used tables. This flag needs to be applied to both
+ <codeph>impalad</codeph> and <codeph>catalogd</codeph>.</li>
+ </ul>
+ <p>Automatic invalidation of metadata provides more stability with lower
+ chances of running out of memory, but the feature could potentially
+ cause performance risks.</p>
+ <note>This is a preview feature in Impala 3.1 and not generally
+ available.</note>
+ </conbody>
+ </concept>
</concept>