You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2020/05/25 13:47:17 UTC
[impala] branch master updated (dd36732 -> 2b9c0cc)
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.
from dd36732 IMPALA-9776: Fix test failure introduced by IMPALA-8834
new 11a1ee5 IMPALA-9772: [DOCS] replace mem_limit texts with revised contents
new 2b9c0cc IMPALA-9743: Fix ExprCardinalityTest may not be marked as in test env
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/topics/impala_mem_limit.xml | 128 +++++++++------------
.../impala/analysis/ExprCardinalityTest.java | 7 ++
2 files changed, 62 insertions(+), 73 deletions(-)
[impala] 02/02: IMPALA-9743: Fix ExprCardinalityTest may not be
marked as in test env
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 2b9c0cc7268994f7fd168897e85525f66ada5c25
Author: stiga-huang <hu...@gmail.com>
AuthorDate: Sat May 23 18:34:43 2020 +0800
IMPALA-9743: Fix ExprCardinalityTest may not be marked as in test env
ExprCardinalityTest should be run with RuntimeEnv.INSTANCE.isTestEnv()
being true. Otherwise, it will leave the test catalog in a bad shape and
fail subsequent tests. The key point it depends on this flag is in
HdfsTable.resetPartitions(), where isStoredInImpaladCatalogCache()
returns false if the flag is not set and then partitionValuesMap_ of the
HdfsTable becomes empty. The test catalog is shared across some tests.
Any other tests that depend on this partitionValuesMap will fail with an
IndexOutOfBoundsException.
Maven tests are run in undetermined orders. Many FE tests will set this
flag, i.e. explicitly call RuntimeEnv.INSTANCE.setTestEnv(true). It’s
safe if any of these tests run before ExprCardinalityTest. However, in
some Jenkins jobs, somehow ExprCardinalityTest is run earlier than any
of these tests, which causes the failures.
Tests:
- Manually run ExprCardinalityTest before an impacted FE test, e.g.
(pushd fe && mvn test -Dtest=ExprCardinalityTest#testMetadata,AnalyzeDDLTest)
Change-Id: I5806dea77eb8e10ccf8ac749b7256603bd79cd97
Reviewed-on: http://gerrit.cloudera.org:8080/15982
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
.../test/java/org/apache/impala/analysis/ExprCardinalityTest.java | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java b/fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java
index b30db8f..28b58da 100644
--- a/fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java
+++ b/fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java
@@ -33,7 +33,9 @@ import org.apache.impala.common.AnalysisSessionFixture;
import org.apache.impala.common.ImpalaException;
import org.apache.impala.common.InternalException;
import org.apache.impala.common.QueryFixture.SelectFixture;
+import org.apache.impala.common.RuntimeEnv;
import org.apache.impala.planner.CardinalityTest;
+import org.junit.BeforeClass;
import org.junit.Test;
/**
@@ -77,6 +79,11 @@ import org.junit.Test;
public class ExprCardinalityTest {
private static AnalysisSessionFixture session_ = new AnalysisSessionFixture();
+ @BeforeClass
+ public static void setUp() {
+ RuntimeEnv.INSTANCE.setTestEnv(true);
+ }
+
private void verifyTableCol(Table table, String colName,
long expectedNdv, long expectedNullCount) {
Column col = table.getColumn(colName);
[impala] 01/02: IMPALA-9772: [DOCS] replace mem_limit texts with
revised contents
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 11a1ee55b0881104b0076880fc818a4cd83fe18a
Author: Shajini Thayasingh <st...@cloudera.com>
AuthorDate: Thu May 21 12:06:58 2020 -0700
IMPALA-9772: [DOCS] replace mem_limit texts with revised contents
explained what will happen if the query process exceeds the mem limit
changed the error message for the last query
also added reference to admission control
Change-Id: I7042b0d66c376e3179fb425109f9103710a25c49
Reviewed-on: http://gerrit.cloudera.org:8080/15973
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Bikramjeet Vig <bi...@cloudera.com>
---
docs/topics/impala_mem_limit.xml | 128 +++++++++++++++++----------------------
1 file changed, 55 insertions(+), 73 deletions(-)
diff --git a/docs/topics/impala_mem_limit.xml b/docs/topics/impala_mem_limit.xml
index 4fed1e1..aec76e6 100644
--- a/docs/topics/impala_mem_limit.xml
+++ b/docs/topics/impala_mem_limit.xml
@@ -19,36 +19,28 @@ under the License.
-->
<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
<concept id="mem_limit">
-
- <title>MEM_LIMIT Query Option</title>
-
- <titlealts audience="PDF">
-
- <navtitle>MEM_LIMIT</navtitle>
-
- </titlealts>
-
- <prolog>
- <metadata>
- <data name="Category" value="Impala"/>
- <data name="Category" value="Impala Query Options"/>
- <data name="Category" value="Scalability"/>
- <data name="Category" value="Memory"/>
- <data name="Category" value="Troubleshooting"/>
- <data name="Category" value="Developers"/>
- <data name="Category" value="Data Analysts"/>
- </metadata>
- </prolog>
-
- <conbody>
-
- <p>
+ <title>MEM_LIMIT Query Option</title>
+ <titlealts audience="PDF">
+ <navtitle>MEM_LIMIT</navtitle>
+ </titlealts>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Impala"/>
+ <data name="Category" value="Impala Query Options"/>
+ <data name="Category" value="Scalability"/>
+ <data name="Category" value="Memory"/>
+ <data name="Category" value="Troubleshooting"/>
+ <data name="Category" value="Developers"/>
+ <data name="Category" value="Data Analysts"/>
+ </metadata>
+ </prolog>
+ <conbody>
+ <p>
The MEM_LIMIT query option defines the maximum amount of memory a query can allocate on
each node. The total memory that can be used by a query is the <codeph>MEM_LIMIT</codeph>
times the number of nodes.
</p>
-
- <p rev="">
+ <p rev="">
There are two levels of memory limit for Impala. The
<codeph>‑‑mem_limit</codeph> startup option sets an overall limit for the
<cmdname>impalad</cmdname> process (which handles multiple queries concurrently). That
@@ -60,62 +52,56 @@ under the License.
with 100GB of memory, the memory available is 100GB or the Docker container's memory
limit, whichever is less.
</p>
-
- <p rev="">
+ <p rev="">
The <codeph>MEM_LIMIT</codeph> query option, which you set through
<cmdname>impala-shell</cmdname> or the <codeph>SET</codeph> statement in a JDBC or ODBC
application, applies to each individual query. The <codeph>MEM_LIMIT</codeph> query option
is usually expressed as a fixed size such as <codeph>10gb</codeph>, and must always be
less than the <cmdname>impalad</cmdname> memory limit.
</p>
-
- <p rev="">
- If query processing exceeds the specified memory limit on any node, either the per-query
- limit or the <cmdname>impalad</cmdname> limit, Impala cancels the query automatically.
- Memory limits are checked periodically during query processing, so the actual memory in
- use might briefly exceed the limit without the query being cancelled.
+ <p rev="">
+ If query processing approaches the specified memory limit on any node, either the
+ per-query limit or the impalad limit, then the SQL operations will start to reduce
+ their memory consumption, for example by writing the temporary data to disk (known as spilling to disk).
+ The result is a query that completes successfully, rather than failing with an out-of-memory error.
+ The tradeoff is decreased performance due to the extra disk I/O to write the temporary data and
+ read it back in. The slowdown could potentially be significant. Thus, while this feature improves
+ reliability, you should optimize your queries, system parameters, and hardware configuration to
+ make this spilling a rare occurrence.
</p>
-
- <p>
- <b>Type:</b> numeric
+ <p>
+ <b>Type:</b> numeric
</p>
-
- <p rev="">
- <b>Units:</b> A numeric argument represents memory size in bytes; you can also use a
+ <p rev="">
+ <b>Units:</b> A numeric argument represents memory size in bytes; you can also use a
suffix of <codeph>m</codeph> or <codeph>mb</codeph> for megabytes, or more commonly
<codeph>g</codeph> or <codeph>gb</codeph> for gigabytes. If you specify a value with
unrecognized formats, subsequent queries fail with an error.
</p>
-
- <p rev="">
- <b>Default:</b> 0 (unlimited)
+ <p rev="">
+ <b>Default:</b> 0 (unlimited)
</p>
-
- <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
-
- <p rev="">
- The <codeph>MEM_LIMIT</codeph> setting is primarily useful in a high-concurrency setting,
- or on a cluster with a workload shared between Impala and other data processing
- components. You can prevent any query from accidentally using much more memory than
- expected, which could negatively impact other Impala queries.
+ <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+ <p rev="">
+ The <codeph>MEM_LIMIT</codeph> setting is primarily useful for production workloads.
+ Impala's Admission Controller can be configured to automatically assign memory limits to
+ queries and limit memory consumption of resource pools. See <xref href="impala_admission.xml#admission_concurrency"/>
+ and <xref href="impala_admission.xml#admission_memory"/> for more information on configuring
+ the resource usage through admission control.
</p>
-
- <p rev="">
+ <p rev="">
Use the output of the <codeph>SUMMARY</codeph> command in <cmdname>impala-shell</cmdname>
to get a report of memory used for each phase of your most heavyweight queries on each
node, and then set a <codeph>MEM_LIMIT</codeph> somewhat higher than that. See
<xref href="impala_explain_plan.xml#perf_summary"/> for usage information about the
<codeph>SUMMARY</codeph> command.
</p>
-
- <p conref="../shared/impala_common.xml#common/example_blurb" rev=""/>
-
- <p rev="">
+ <p conref="../shared/impala_common.xml#common/example_blurb" rev=""/>
+ <p rev="">
The following examples show how to set the <codeph>MEM_LIMIT</codeph> query option using a
fixed number of bytes, or suffixes representing gigabytes or megabytes.
</p>
-
-<codeblock rev="">
+ <codeblock rev="">
[localhost:21000] > set mem_limit=3000000000;
MEM_LIMIT set to 3000000000
[localhost:21000] > select 5;
@@ -160,13 +146,11 @@ MEM_LIMIT set to 3mb
| 5 |
+---+
</codeblock>
-
- <p rev="">
+ <p rev="">
The following examples show how unrecognized <codeph>MEM_LIMIT</codeph> values lead to
errors for subsequent queries.
</p>
-
-<codeblock rev="">
+ <codeblock rev="">
[localhost:21000] > set mem_limit=3pb;
MEM_LIMIT set to 3pb
[localhost:21000] > select 5;
@@ -178,16 +162,14 @@ MEM_LIMIT set to xyz
Query: select 5
ERROR: Failed to parse query memory limit from 'xyz'.
</codeblock>
-
- <p rev="">
+ <p rev="">
The following examples shows the automatic query cancellation when the
<codeph>MEM_LIMIT</codeph> value is exceeded on any host involved in the Impala query.
First it runs a successful query and checks the largest amount of memory used on any node
for any stage of the query. Then it sets an artificially low <codeph>MEM_LIMIT</codeph>
setting so that the same query cannot run.
</p>
-
-<codeblock rev="">
+ <codeblock rev="">
[localhost:21000] > select count(*) from customer;
Query: select count(*) from customer
+----------+
@@ -222,10 +204,10 @@ MEM_LIMIT set to 15mb
[localhost:21000] > select count(distinct c_name) from customer;
Query: select count(distinct c_name) from customer
ERROR:
-Memory limit exceeded
-Query did not have enough memory to get the minimum required buffers in the block manager.
-</codeblock>
-
- </conbody>
-
+Rejected query from pool default-pool: minimum memory reservation is greater than memory available to the query
+for buffer reservations. Memory reservation needed given the current plan: 38.00 MB. Adjust either the mem_limit
+or the pool config (max-query-mem-limit, min-query-mem-limit) for the query to allow the query memory limit to be
+at least 70.00 MB. Note that changing the mem_limit may also change the plan. See the query profile for more
+information about the per-node memory requirements.</codeblock>
+ </conbody>
</concept>