You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jr...@apache.org on 2017/07/10 23:09:08 UTC
[2/3] incubator-impala git commit: [DOCS] Advise setting
vm.overcommit_memory=1 in various places
[DOCS] Advise setting vm.overcommit_memory=1 in various places
Reusing the same advice under "Known Issues", scalability
considerations, and in the Impala + Kerberos section.
Change-Id: Icbfa755e2c9769a8458fd93362769856cf32e301
Reviewed-on: http://gerrit.cloudera.org:8080/7349
Reviewed-by: Mostafa Mokhtar <mm...@cloudera.com>
Tested-by: Impala Public Jenkins
Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/db3f3235
Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/db3f3235
Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/db3f3235
Branch: refs/heads/master
Commit: db3f3235d7c28ff9963e0332b830ecc86f54a505
Parents: cd44381
Author: John Russell <jr...@cloudera.com>
Authored: Fri Jun 30 16:00:30 2017 -0700
Committer: Impala Public Jenkins <im...@gerrit.cloudera.org>
Committed: Mon Jul 10 23:06:45 2017 +0000
----------------------------------------------------------------------
docs/shared/impala_common.xml | 32 ++++++++++++++++++++++++++++++++
docs/topics/impala_kerberos.xml | 8 ++++++++
docs/topics/impala_known_issues.xml | 11 +++++++++++
docs/topics/impala_scalability.xml | 8 ++++++++
4 files changed, 59 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/db3f3235/docs/shared/impala_common.xml
----------------------------------------------------------------------
diff --git a/docs/shared/impala_common.xml b/docs/shared/impala_common.xml
index 6e65c40..2ec2537 100644
--- a/docs/shared/impala_common.xml
+++ b/docs/shared/impala_common.xml
@@ -1365,6 +1365,38 @@ select c_first_name, c_last_name from customer where lower(trim(c_last_name)) rl
<codeph>user@example.com</codeph>, in a Kerberized environment.
</p>
+ <p id="vm_overcommit_memory_intro">
+ On a kerberized cluster with high memory utilization, <cmdname>kinit</cmdname> commands executed after
+ every <codeph>'kerberos_reinit_interval'</codeph> may cause out-of-memory errors, because executing
+ the command involves a fork of the Impala process. The error looks similar to the following:
+<codeblock><![CDATA[
+Failed to obtain Kerberos ticket for principal: <varname>principal_details</varname>
+Failed to execute shell cmd: 'kinit -k -t <varname>keytab_details</varname>',
+error was: Error(12): Cannot allocate memory
+]]>
+</codeblock>
+ </p>
+
+ <p id="vm_overcommit_memory_start">
+ The following command changes the <codeph>vm.overcommit_memory</codeph>
+ setting immediately on a running host. However, this setting is reset
+ when the host is restarted.
+<codeblock><![CDATA[
+echo 1 > /proc/sys/vm/overcommit_memory
+]]>
+</codeblock>
+ </p>
+ <p>
+ To change the setting in a persistent way, add the following line to the
+ <filepath>/etc/sysctl.conf</filepath> file:
+<codeblock><![CDATA[
+vm.overcommit_memory=1
+]]>
+</codeblock>
+ </p>
+ <p id="vm_overcommit_memory_end">
+ Then run <codeph>sysctl -p</codeph>. No reboot is needed.
+ </p>
<ul>
<li id="grant_revoke_single">
Currently, each Impala <codeph>GRANT</codeph> or <codeph>REVOKE</codeph> statement can only grant or
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/db3f3235/docs/topics/impala_kerberos.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_kerberos.xml b/docs/topics/impala_kerberos.xml
index 5032fba..5d97aeb 100644
--- a/docs/topics/impala_kerberos.xml
+++ b/docs/topics/impala_kerberos.xml
@@ -351,4 +351,12 @@ $ chown impala:impala impala-http.keytab</codeblock>
</conbody>
</concept>
+ <concept rev="IMPALA-2294" id="kerberos_overhead_memory_usage">
+ <title>Kerberos-Related Memory Overhead for Large Clusters</title>
+ <conbody>
+ <p conref="../shared/impala_common.xml#common/vm_overcommit_memory_intro"/>
+ <p conref="../shared/impala_common.xml#common/vm_overcommit_memory_start" conrefend="vm_overcommit_memory_end"/>
+ </conbody>
+ </concept>
+
</concept>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/db3f3235/docs/topics/impala_known_issues.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_known_issues.xml b/docs/topics/impala_known_issues.xml
index 30cc245..51920c9 100644
--- a/docs/topics/impala_known_issues.xml
+++ b/docs/topics/impala_known_issues.xml
@@ -623,6 +623,17 @@ impala hard nproc 262144
</conbody>
</concept>
+ <concept id="IMPALA-2294">
+ <title>Kerberos initialization errors due to high memory usage</title>
+ <conbody>
+ <p conref="../shared/impala_common.xml#common/vm_overcommit_memory_intro"/>
+ <p><b>Bug:</b> <xref keyref="IMPALA-2294">IMPALA-2294</xref></p>
+ <p><b>Severity:</b> High</p>
+ <p><b>Workaround:</b></p>
+ <p conref="../shared/impala_common.xml#common/vm_overcommit_memory_start" conrefend="vm_overcommit_memory_end"/>
+ </conbody>
+ </concept>
+
<concept id="drop_table_purge_s3a">
<title>DROP TABLE PURGE on S3A table may not delete externally written files</title>
<conbody>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/db3f3235/docs/topics/impala_scalability.xml
----------------------------------------------------------------------
diff --git a/docs/topics/impala_scalability.xml b/docs/topics/impala_scalability.xml
index af61e7a..2ea7603 100644
--- a/docs/topics/impala_scalability.xml
+++ b/docs/topics/impala_scalability.xml
@@ -887,6 +887,14 @@ so other secure services might be affected temporarily.
</conbody>
</concept>
+ <concept rev="IMPALA-2294" id="kerberos_overhead_memory_usage">
+ <title>Kerberos-Related Memory Overhead for Large Clusters</title>
+ <conbody>
+ <p conref="../shared/impala_common.xml#common/vm_overcommit_memory_intro"/>
+ <p conref="../shared/impala_common.xml#common/vm_overcommit_memory_start" conrefend="vm_overcommit_memory_end"/>
+ </conbody>
+ </concept>
+
<concept id="scalability_hotspots" rev="2.5.0 IMPALA-2696">
<title>Avoiding CPU Hotspots for HDFS Cached Data</title>
<conbody>