You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by mi...@apache.org on 2023/10/02 17:19:39 UTC
[impala] branch master updated (2d3289027 -> ae14d78c8)
This is an automated email from the ASF dual-hosted git repository.
michaelsmith pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
from 2d3289027 IMPALA-12406: OPTIMIZE statement as an alias for INSERT OVERWRITE
new 78b9285da IMPALA-12461 part1: Avoid taking db/table level locks db/table self-event check
new 782cda449 IMPALA-12453: Increase yarn high disk threshold
new 80af98645 IMPALA-12469: Allow short if statements on a single line in clang-format
new 89a48b80a IMPALA-12444: Fix minimum parallelism bug in scan fragment
new ae14d78c8 IMPALA-12464: Add opens for java.util.regex
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.clang-format | 2 +
be/src/common/init.cc | 3 +-
.../java/org/apache/impala/analysis/Analyzer.java | 2 +
.../impala/catalog/CatalogServiceCatalog.java | 134 ++++++++++++---------
fe/src/main/java/org/apache/impala/catalog/Db.java | 17 ++-
.../java/org/apache/impala/catalog/HdfsTable.java | 1 +
.../main/java/org/apache/impala/catalog/Table.java | 42 ++++---
.../org/apache/impala/planner/CostingSegment.java | 2 +-
.../org/apache/impala/planner/PlanFragment.java | 24 ++--
.../java/org/apache/impala/planner/ScanNode.java | 37 +++---
.../java/org/apache/impala/service/Frontend.java | 28 ++---
.../common/etc/hadoop/conf/yarn-site.xml.py | 5 +
tests/custom_cluster/test_executor_groups.py | 57 +++------
13 files changed, 186 insertions(+), 168 deletions(-)
[impala] 01/05: IMPALA-12461 part1: Avoid taking db/table level locks db/table self-event check
Posted by mi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
michaelsmith pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 78b9285da457c6853e513f3852730867d4dbe632
Author: Csaba Ringhofer <cs...@cloudera.com>
AuthorDate: Wed Sep 27 21:22:31 2023 +0200
IMPALA-12461 part1: Avoid taking db/table level locks db/table self-event check
DB / Table level locks can be held for a long time by DDL/DML
operations in the catalogd. Trying to get the lock during self-event
check would mean blocking if there is an ongoing operation for
the given db/table.
This patch tries to solve it for the easy case of db/table level
events by using self-event specific locking that should be only
taken for short times to add/remove events from/to the in-flight
event list. The InFlightEvents object itself is used as lock
and no other of catalogd's lock must be acquired after it to
avoid deadlock.
Postponing solving this for partition level events as that would
be more complex, as both the partition list and the partitions'
in-flight event lists would need to be protected from parallel
operations that add/remove partitions.
Testing:
- ran event processing related tests
Change-Id: Ife455de09ab2e262bde1e4b5bd54c8c54c75f2cd
Reviewed-on: http://gerrit.cloudera.org:8080/20516
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
.../impala/catalog/CatalogServiceCatalog.java | 134 ++++++++++++---------
fe/src/main/java/org/apache/impala/catalog/Db.java | 17 ++-
.../java/org/apache/impala/catalog/HdfsTable.java | 1 +
.../main/java/org/apache/impala/catalog/Table.java | 42 ++++---
4 files changed, 114 insertions(+), 80 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java b/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
index ddb08e75f..27d7a64e4 100644
--- a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
+++ b/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
@@ -1046,83 +1046,99 @@ public class CatalogServiceCatalog extends Catalog {
LOG.debug("Not a self-event because eventId is {}", versionNumber);
return false;
}
- Db db = getDb(ctx.getDbName());
+ Db db = getDb(ctx.getDbName()); // Uses thread-safe hashmap.
if (db == null) {
throw new DatabaseNotFoundException("Database " + ctx.getDbName() + " not found");
}
- // if the given tblName is null we look db's in-flight events
if (ctx.getTblName() == null) {
- //TODO use read/write locks for both table and db
- if (!tryLockDb(db)) {
- throw new CatalogException("Could not acquire lock on database object " +
- db.getName());
- }
- versionLock_.writeLock().unlock();
- try {
- boolean removed = db.removeFromVersionsForInflightEvents(versionNumber);
- if (!removed) {
- LOG.debug("Could not find version {} in the in-flight event list of database "
- + "{}", versionNumber, db.getName());
- }
- return removed;
- } finally {
- db.getLock().unlock();
- }
+ // Not taking lock, rely on Db's internal locking.
+ return evaluateSelfEventForDb(db, versionNumber);
}
- Table tbl = db.getTable(ctx.getTblName());
+
+ Table tbl = db.getTable(ctx.getTblName()); // Uses thread-safe hashmap.
if (tbl == null) {
throw new TableNotFoundException(
String.format("Table %s.%s not found", ctx.getDbName(), ctx.getTblName()));
}
- // we should acquire the table lock so that we wait for any other updates
- // happening to this table at the same time
+
+ if (ctx.getPartitionKeyValues() == null) {
+ // Not taking lock, rely on Table's internal locking.
+ return evaluateSelfEventForTable(tbl, isInsertEvent, versionNumber);
+ }
+
+ // TODO: Could be a Precondition? If partitionKeyValues != null, this should be
+ // a HDFS table.
+ if (!(tbl instanceof HdfsTable)) return false;
+
+ // We should acquire the table lock so that we wait for any other updates
+ // happening to this table at the same time, as they could affect the list of
+ // partitions.
+ // TODO: add more fine grained locking to protect partitions without taking
+ // longly held locks (IMPALA-12461)
+ // Increasing the timeout would not help if the table lock is held by
+ // a concurrent DDL as we would need the lock during the actual processing of the
+ // event.
if (!tryWriteLock(tbl)) {
throw new CatalogException(String.format("Error during self-event evaluation "
+ "for table %s due to lock contention", tbl.getFullName()));
}
versionLock_.writeLock().unlock();
try {
- List<List<TPartitionKeyValue>> partitionKeyValues = ctx.getPartitionKeyValues();
- // if the partitionKeyValues is null, we look for tbl's in-flight events
- if (partitionKeyValues == null) {
- boolean removed =
- tbl.removeFromVersionsForInflightEvents(isInsertEvent, versionNumber);
- if (!removed) {
- LOG.debug("Could not find {} {} in in-flight event list of table {}",
- isInsertEvent ? "eventId" : "version", versionNumber, tbl.getFullName());
- }
- return removed;
- }
- if (tbl instanceof HdfsTable) {
- List<String> failingPartitions = new ArrayList<>();
- int len = partitionKeyValues.size();
- for (int i=0; i<len; ++i) {
- List<TPartitionKeyValue> partitionKeyValue = partitionKeyValues.get(i);
- versionNumber = isInsertEvent ? ctx.getInsertEventId(i) : versionNumber;
- HdfsPartition hdfsPartition =
- ((HdfsTable) tbl).getPartitionFromThriftPartitionSpec(partitionKeyValue);
- if (hdfsPartition == null
- || !hdfsPartition.removeFromVersionsForInflightEvents(isInsertEvent,
- versionNumber)) {
- // even if this is an error condition we should not bail out early since we
- // should clean up the self-event state on the rest of the partitions
- String partName = HdfsTable.constructPartitionName(partitionKeyValue);
- if (hdfsPartition == null) {
- LOG.debug("Partition {} not found during self-event "
- + "evaluation for the table {}", partName, tbl.getFullName());
- } else {
- LOG.trace("Could not find {} in in-flight event list of the partition {} "
- + "of table {}", versionNumber, partName, tbl.getFullName());
- }
- failingPartitions.add(partName);
- }
- }
- return failingPartitions.isEmpty();
- }
+ return evaluateSelfEventForPartition(
+ ctx, (HdfsTable)tbl, isInsertEvent, versionNumber);
} finally {
tbl.releaseWriteLock();
}
- return false;
+ }
+
+ private boolean evaluateSelfEventForDb(Db db, long versionNumber)
+ throws CatalogException {
+ boolean removed = db.removeFromVersionsForInflightEvents(versionNumber);
+ if (!removed) {
+ LOG.debug("Could not find version {} in the in-flight event list of database "
+ + "{}", versionNumber, db.getName());
+ }
+ return removed;
+ }
+
+ private boolean evaluateSelfEventForTable(Table tbl,
+ boolean isInsertEvent, long versionNumber) throws CatalogException {
+ boolean removed =
+ tbl.removeFromVersionsForInflightEvents(isInsertEvent, versionNumber);
+ if (!removed) {
+ LOG.debug("Could not find {} {} in in-flight event list of table {}",
+ isInsertEvent ? "eventId" : "version", versionNumber, tbl.getFullName());
+ }
+ return removed;
+ }
+
+ private boolean evaluateSelfEventForPartition(SelfEventContext ctx, HdfsTable tbl,
+ boolean isInsertEvent, long versionNumber) throws CatalogException {
+ List<List<TPartitionKeyValue>> partitionKeyValues = ctx.getPartitionKeyValues();
+ List<String> failingPartitions = new ArrayList<>();
+ int len = partitionKeyValues.size();
+ for (int i=0; i<len; ++i) {
+ List<TPartitionKeyValue> partitionKeyValue = partitionKeyValues.get(i);
+ versionNumber = isInsertEvent ? ctx.getInsertEventId(i) : versionNumber;
+ HdfsPartition hdfsPartition =
+ tbl.getPartitionFromThriftPartitionSpec(partitionKeyValue);
+ if (hdfsPartition == null
+ || !hdfsPartition.removeFromVersionsForInflightEvents(isInsertEvent,
+ versionNumber)) {
+ // even if this is an error condition we should not bail out early since we
+ // should clean up the self-event state on the rest of the partitions
+ String partName = HdfsTable.constructPartitionName(partitionKeyValue);
+ if (hdfsPartition == null) {
+ LOG.debug("Partition {} not found during self-event "
+ + "evaluation for the table {}", partName, tbl.getFullName());
+ } else {
+ LOG.trace("Could not find {} in in-flight event list of the partition {} "
+ + "of table {}", versionNumber, partName, tbl.getFullName());
+ }
+ failingPartitions.add(partName);
+ }
+ }
+ return failingPartitions.isEmpty();
}
/**
diff --git a/fe/src/main/java/org/apache/impala/catalog/Db.java b/fe/src/main/java/org/apache/impala/catalog/Db.java
index 0cb800b0c..ad25324fb 100644
--- a/fe/src/main/java/org/apache/impala/catalog/Db.java
+++ b/fe/src/main/java/org/apache/impala/catalog/Db.java
@@ -99,6 +99,9 @@ public class Db extends CatalogObjectImpl implements FeDb {
private boolean isSystemDb_ = false;
// tracks the in-flight metastore events for this db
+ // Also used as a monitor object to synchronize access to it to avoid blocking on table
+ // lock during self-event check. If both this and dbLock_ or catalog version lock are
+ // taken, inFlightEvents_ must be the last to avoid deadlock.
private final InFlightEvents inFlightEvents_ = new InFlightEvents();
// lock to make sure modifications to the Db object are atomically done along with
@@ -565,10 +568,9 @@ public class Db extends CatalogObjectImpl implements FeDb {
* @return true if version was successfully removed, false if didn't exist
*/
public boolean removeFromVersionsForInflightEvents(long versionNumber) {
- Preconditions.checkState(dbLock_.isHeldByCurrentThread(),
- "removeFromVersionsForInflightEvents called without getting the db lock for "
- + getName() + " database.");
- return inFlightEvents_.remove(false, versionNumber);
+ synchronized (inFlightEvents_) {
+ return inFlightEvents_.remove(false, versionNumber);
+ }
}
/**
@@ -579,10 +581,15 @@ public class Db extends CatalogObjectImpl implements FeDb {
* @param versionNumber version number to add
*/
public void addToVersionsForInflightEvents(long versionNumber) {
+ // The lock is not needed for thread safety, just verifying existing behavior.
Preconditions.checkState(dbLock_.isHeldByCurrentThread(),
"addToVersionsForInFlightEvents called without getting the db lock for "
+ getName() + " database.");
- if (!inFlightEvents_.add(false, versionNumber)) {
+ boolean added = false;
+ synchronized (inFlightEvents_) {
+ added = inFlightEvents_.add(false, versionNumber);
+ }
+ if (!added) {
LOG.warn(String.format("Could not add version %s to the list of in-flight "
+ "events. This could cause unnecessary database %s invalidation when the "
+ "event is processed", versionNumber, getName()));
diff --git a/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java b/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
index c57671a43..8c072844d 100644
--- a/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
@@ -2017,6 +2017,7 @@ public class HdfsTable extends Table implements FeFsTable {
return ret;
}
+ // TODO: why is there a single synchronized function?
@Override
protected synchronized void loadFromThrift(TTable thriftTable)
throws TableLoadingException {
diff --git a/fe/src/main/java/org/apache/impala/catalog/Table.java b/fe/src/main/java/org/apache/impala/catalog/Table.java
index 2364f82f9..3f95764c1 100644
--- a/fe/src/main/java/org/apache/impala/catalog/Table.java
+++ b/fe/src/main/java/org/apache/impala/catalog/Table.java
@@ -148,8 +148,11 @@ public abstract class Table extends CatalogObjectImpl implements FeTable {
protected volatile long createEventId_ = -1;
// tracks the in-flight metastore events for this table. Used by Events processor to
- // avoid unnecessary refresh when the event is received
- private final InFlightEvents inFlightEvents = new InFlightEvents();
+ // avoid unnecessary refresh when the event is received.
+ // Also used as a monitor object to synchronize access to it to avoid blocking on table
+ // lock during self-event check. If both this and tableLock_ or catalog version lock are
+ // taken, inFlightEvents_ must be the last to avoid deadlock.
+ private final InFlightEvents inFlightEvents_ = new InFlightEvents();
// Table metrics. These metrics are applicable to all table types. Each subclass of
// Table can define additional metrics specific to that table type.
@@ -981,12 +984,13 @@ public abstract class Table extends CatalogObjectImpl implements FeTable {
*/
public boolean removeFromVersionsForInflightEvents(
boolean isInsertEvent, long versionNumber) {
- Preconditions.checkState(isWriteLockedByCurrentThread(),
- "removeFromVersionsForInFlightEvents called without taking the table lock on "
- + getFullName());
- boolean removed = inFlightEvents.remove(isInsertEvent, versionNumber);
- if (removed) {
- metrics_.getCounter(NUMBER_OF_INFLIGHT_EVENTS).dec();
+ boolean removed = false;
+ synchronized (inFlightEvents_) {
+ removed = inFlightEvents_.remove(isInsertEvent, versionNumber);
+ // Locked updating of counters is not need for correctnes but tests may rely on it.
+ if (removed) {
+ metrics_.getCounter(NUMBER_OF_INFLIGHT_EVENTS).dec();
+ }
}
return removed;
}
@@ -1004,16 +1008,22 @@ public abstract class Table extends CatalogObjectImpl implements FeTable {
* capacity
*/
public void addToVersionsForInflightEvents(boolean isInsertEvent, long versionNumber) {
- // we generally don't take locks on Incomplete tables since they are atomically
- // replaced during load
+ // We generally don't take locks on Incomplete tables since they are atomically
+ // replaced during load.
+ // The lock is not needed for thread safety, just verifying existing behavior.
Preconditions.checkState(
this instanceof IncompleteTable || isWriteLockedByCurrentThread());
- if (!inFlightEvents.add(isInsertEvent, versionNumber)) {
- LOG.warn(String.format("Could not add %s version to the table %s. This could "
- + "cause unnecessary refresh of the table when the event is received by the "
- + "Events processor.", versionNumber, getFullName()));
- } else {
- metrics_.getCounter(NUMBER_OF_INFLIGHT_EVENTS).inc();
+ boolean added = false;
+ synchronized (inFlightEvents_) {
+ added = inFlightEvents_.add(isInsertEvent, versionNumber);
+ // Locked updating of counters is not need for correctnes but tests may rely on it.
+ if (!added) {
+ LOG.warn(String.format("Could not add %s version to the table %s. This could "
+ + "cause unnecessary refresh of the table when the event is received by the "
+ + "Events processor.", versionNumber, getFullName()));
+ } else {
+ metrics_.getCounter(NUMBER_OF_INFLIGHT_EVENTS).inc();
+ }
}
}
[impala] 02/05: IMPALA-12453: Increase yarn high disk threshold
Posted by mi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
michaelsmith pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 782cda44996ac73c0c14cc0727683c9e3ef1fd1a
Author: Michael Smith <mi...@cloudera.com>
AuthorDate: Thu Sep 21 15:17:07 2023 -0700
IMPALA-12453: Increase yarn high disk threshold
Yarn will mark a node as unavailable for running jobs if it exceeds the
disk threshold. We only have one node, so that means jobs hang until
disk frees up. Exhaustive test runs are approaching using 300GB, which
is a common size for our test environments, and we see some tests
hanging on yarn jobs.
Increases the disk threshold from default of 90% to 99%; in test
environments the remaining 1% is ~3GB. That should still be sufficient
in case yarn needs to throttle workloads to avoid using all the disk,
while leaving most of the disk available.
Change-Id: I06b723e4ecea94ce95fb2bd41bd356fba694e87a
Reviewed-on: http://gerrit.cloudera.org:8080/20501
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
.../cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py b/testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py
index f9372bf5a..dbcd9f9be 100644
--- a/testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py
+++ b/testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py
@@ -71,6 +71,11 @@ CONFIG = {
# to speed up data-loading.
'yarn.nodemanager.resource.memory-mb': _get_yarn_nm_ram_mb(),
+ # Allow YARN to run with at least 3GB disk free. Otherwise it hangs completely.
+ # Avoids disabling YARN disk monitoring completely because otherwise multiple jobs might
+ # use up all the disk in a scenario where otherwise they could complete sequentially.
+ 'yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage': 99,
+
# Increase YARN container resources to 2GB to avoid dataload failures
'yarn.app.mapreduce.am.resource.mb': 2048,
[impala] 04/05: IMPALA-12444: Fix minimum parallelism bug in scan fragment
Posted by mi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
michaelsmith pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 89a48b80a2689115d6bd90bf4312a452ae11477a
Author: Riza Suminto <ri...@cloudera.com>
AuthorDate: Thu Aug 24 11:30:37 2023 -0700
IMPALA-12444: Fix minimum parallelism bug in scan fragment
Scan fragment did not follow PROCESSING_COST_MIN_THREADS set by user
even if total scan ranges allow to do so. This patch fix the issue by
exposing ScanNode.maxScannerThreads_ to
PlanFragment.adjustToMaxParallelism(). By using
ScanNode.maxScannerThreads_ as an upper bound, ScanNode does not need to
artificially lower ProcessingCost if maxScannerThreads_ is lower than
minimum parallelism dictated by the original ProcessingCost. Thus, the
synthetic ProcessingCost logic in ScanNode class is revised to only
apply if input cardinality is unknown (-1).
This patch also does the following adjustments:
- Remove some dead codes in Frontend.java and PlanFragment.java.
- Add sanity check such that PROCESSING_COST_MIN_THREADS <=
MAX_FRAGMENT_INSTANCES_PER_NODE.
- Tidy up test_query_cpu_count_divisor_default to reduce number of
SET query.
Testing:
- Update test_query_cpu_count_divisor_default to ensure that
PROCESSING_COST_MIN_THREADS is respected by scan fragment and error
is returned if PROCESSING_COST_MIN_THREADS is greater than
MAX_FRAGMENT_INSTANCES_PER_NODE.
- Pass test_executor_groups.py.
Change-Id: I69e5a80146d4ac41de5ef406fc2bdceffe3ec394
Reviewed-on: http://gerrit.cloudera.org:8080/20475
Reviewed-by: Kurt Deschler <kd...@cloudera.com>
Reviewed-by: Wenzhe Zhou <wz...@cloudera.com>
Tested-by: Riza Suminto <ri...@cloudera.com>
---
.../java/org/apache/impala/analysis/Analyzer.java | 2 +
.../org/apache/impala/planner/CostingSegment.java | 2 +-
.../org/apache/impala/planner/PlanFragment.java | 24 ++++-----
.../java/org/apache/impala/planner/ScanNode.java | 37 +++++++-------
.../java/org/apache/impala/service/Frontend.java | 28 +++++------
tests/custom_cluster/test_executor_groups.py | 57 ++++++++--------------
6 files changed, 63 insertions(+), 87 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/analysis/Analyzer.java b/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
index 1e024d525..224be7859 100644
--- a/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
+++ b/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
@@ -580,6 +580,8 @@ public class Analyzer {
// singleton.
private int numExecutorsForPlanning_ = -1;
+ // Number of available cores per executor node.
+ // Set by Frontend.java.
private int availableCoresPerNode_ = -1;
// Cache of KuduTables opened for this query. (map from table name to kudu table)
diff --git a/fe/src/main/java/org/apache/impala/planner/CostingSegment.java b/fe/src/main/java/org/apache/impala/planner/CostingSegment.java
index 27ea37300..fed37bf29 100644
--- a/fe/src/main/java/org/apache/impala/planner/CostingSegment.java
+++ b/fe/src/main/java/org/apache/impala/planner/CostingSegment.java
@@ -192,7 +192,7 @@ public class CostingSegment extends TreeNode<CostingSegment> {
nodeStepCount, minParallelism, maxParallelism, producerCost, cost_);
newParallelism = Math.max(newParallelism, cost_.getNumInstancesExpected());
Preconditions.checkState(newParallelism <= maxParallelism,
- "originalParallelism=" + originalParallelism + ". newParallelism="
+ getRootId() + " originalParallelism=" + originalParallelism + ". newParallelism="
+ newParallelism + " > maxParallelism=" + maxParallelism);
if (LOG.isTraceEnabled()) {
diff --git a/fe/src/main/java/org/apache/impala/planner/PlanFragment.java b/fe/src/main/java/org/apache/impala/planner/PlanFragment.java
index 44b52f152..de708b0a8 100644
--- a/fe/src/main/java/org/apache/impala/planner/PlanFragment.java
+++ b/fe/src/main/java/org/apache/impala/planner/PlanFragment.java
@@ -1062,6 +1062,7 @@ public class PlanFragment extends TreeNode<PlanFragment> {
int parentParallelism, int nodeStepCount) {
int maxThreadAllowed = IntMath.saturatedMultiply(maxThreadPerNode, getNumNodes());
boolean canTryLower = true;
+ int maxScannerThreads = Integer.MAX_VALUE;
// Compute selectedParallelism as the maximum allowed parallelism.
int selectedParallelism = getNumInstances();
@@ -1099,13 +1100,12 @@ public class PlanFragment extends TreeNode<PlanFragment> {
if (!scanNodes.isEmpty()) {
// The existence of scan node may justify an increase of parallelism for this
// union fargment, but it should be capped at costBasedMaxParallelism.
- long maxRangesPerScanNode = 1;
+ maxScannerThreads = 1;
for (ScanNode scanNode : scanNodes) {
- maxRangesPerScanNode =
- Math.max(maxRangesPerScanNode, scanNode.getEffectiveNumScanRanges());
+ maxScannerThreads = Math.max(maxScannerThreads, scanNode.maxScannerThreads_);
}
- maxParallelism_ = Math.max(maxParallelism_,
- (int) Math.min(costBasedMaxParallelism, maxRangesPerScanNode));
+ maxParallelism_ = Math.max(
+ maxParallelism_, Math.min(costBasedMaxParallelism, maxScannerThreads));
}
if (maxParallelism_ > maxThreadAllowed) {
@@ -1135,8 +1135,8 @@ public class PlanFragment extends TreeNode<PlanFragment> {
if (!scanNodes.isEmpty()) {
Preconditions.checkState(scanNodes.size() == 1);
ScanNode scanNode = scanNodes.get(0);
- maxParallelism_ =
- (int) Math.min(maxParallelism_, scanNode.getEffectiveNumScanRanges());
+ maxScannerThreads = scanNode.maxScannerThreads_;
+ maxParallelism_ = Math.min(maxParallelism_, maxScannerThreads);
// Prevent caller from lowering parallelism if fragment has ScanNode
// because there is no child fragment to compare with.
@@ -1153,7 +1153,7 @@ public class PlanFragment extends TreeNode<PlanFragment> {
getNumInstances(), selectedParallelism, "Follow maxThreadPerNode.");
}
} else {
- if (maxParallelism_ < minParallelism && scanNodes.isEmpty()) {
+ if (maxParallelism_ < minParallelism && minParallelism < maxScannerThreads) {
maxParallelism_ = minParallelism;
canTryLower = false;
if (LOG.isTraceEnabled()) {
@@ -1177,7 +1177,7 @@ public class PlanFragment extends TreeNode<PlanFragment> {
// Initialize this fragment's parallelism to the selectedParallelism.
setAdjustedInstanceCount(selectedParallelism);
- return canTryLower;
+ return canTryLower && selectedParallelism > 1;
}
private boolean hasUnionNode() {
@@ -1186,12 +1186,6 @@ public class PlanFragment extends TreeNode<PlanFragment> {
return !nodes.isEmpty();
}
- private boolean hasScanNode() {
- List<ScanNode> nodes = Lists.newArrayList();
- collectPlanNodes(Predicates.instanceOf(ScanNode.class), nodes);
- return !nodes.isEmpty();
- }
-
/**
* Compute {@link CoreCount} of this fragment and populate it into 'fragmentCoreState'.
* @param fragmentCoreState A map holding per-fragment core state.
diff --git a/fe/src/main/java/org/apache/impala/planner/ScanNode.java b/fe/src/main/java/org/apache/impala/planner/ScanNode.java
index a2bcda486..66ef705ce 100644
--- a/fe/src/main/java/org/apache/impala/planner/ScanNode.java
+++ b/fe/src/main/java/org/apache/impala/planner/ScanNode.java
@@ -84,6 +84,10 @@ abstract public class ScanNode extends PlanNode {
// Refer to the comment of 'TableRef.tableNumRowsHint_'
protected long tableNumRowsHint_ = -1;
+ // Maximum number of scanner threads after considering number of scan ranges
+ // and related query options.Calculated at computeScanProcessingCost.
+ protected int maxScannerThreads_ = -1;
+
public ScanNode(PlanNodeId id, TupleDescriptor desc, String displayName) {
super(id, desc.getId().asList(), displayName);
desc_ = desc;
@@ -358,39 +362,38 @@ abstract public class ScanNode extends PlanNode {
*/
protected ProcessingCost computeScanProcessingCost(TQueryOptions queryOptions) {
Preconditions.checkArgument(queryOptions.isCompute_processing_cost());
- ProcessingCost cardinalityBasedCost =
- ProcessingCost.basicCost(getDisplayLabel(), getInputCardinality(),
- ExprUtil.computeExprsTotalCost(conjuncts_), rowMaterializationCost());
// maxThread calculation below intentionally does not include core count from
// executor group config. This is to allow scan fragment parallelism to scale
// regardless of the core count limit.
int maxThreadsPerNode = Math.max(queryOptions.getProcessing_cost_min_threads(),
queryOptions.getMax_fragment_instances_per_node());
- int maxScannerThreads = (int) Math.min(getEffectiveNumScanRanges(),
+ maxScannerThreads_ = (int) Math.min(getEffectiveNumScanRanges(),
IntMath.saturatedMultiply(getNumNodes(), maxThreadsPerNode));
-
- if (getInputCardinality() == 0) {
- Preconditions.checkState(cardinalityBasedCost.getTotalCost() == 0,
- "Scan is empty but cost is non-zero.");
+ long inputCardinality = getInputCardinality();
+
+ if (inputCardinality >= 0) {
+ ProcessingCost cardinalityBasedCost =
+ ProcessingCost.basicCost(getDisplayLabel(), inputCardinality,
+ ExprUtil.computeExprsTotalCost(conjuncts_), rowMaterializationCost());
+ if (inputCardinality == 0) {
+ Preconditions.checkState(cardinalityBasedCost.getTotalCost() == 0,
+ "Scan is empty but cost is non-zero.");
+ }
return cardinalityBasedCost;
- } else if (maxScannerThreads < cardinalityBasedCost.getTotalCost()
- / BackendConfig.INSTANCE.getMinProcessingPerThread()) {
- // Input cardinality is unknown or cost is too high compared to maxScanThreads
- // Return synthetic ProcessingCost based on maxScanThreads.
+ } else {
+ // Input cardinality is unknown. Return synthetic ProcessingCost based on
+ // maxScannerThreads_.
long syntheticCardinality =
- Math.max(1, Math.min(getInputCardinality(), maxScannerThreads));
+ Math.max(1, Math.min(inputCardinality, maxScannerThreads_));
long syntheticPerRowCost = LongMath.saturatedMultiply(
Math.max(1,
BackendConfig.INSTANCE.getMinProcessingPerThread() / syntheticCardinality),
- maxScannerThreads);
+ maxScannerThreads_);
return ProcessingCost.basicCost(
getDisplayLabel(), syntheticCardinality, 0, syntheticPerRowCost);
}
-
- // None of the conditions above apply.
- return cardinalityBasedCost;
}
/**
diff --git a/fe/src/main/java/org/apache/impala/service/Frontend.java b/fe/src/main/java/org/apache/impala/service/Frontend.java
index 2821dfc5e..0d2153656 100644
--- a/fe/src/main/java/org/apache/impala/service/Frontend.java
+++ b/fe/src/main/java/org/apache/impala/service/Frontend.java
@@ -180,6 +180,7 @@ import org.apache.impala.thrift.TGetTableHistoryResult;
import org.apache.impala.thrift.TGetTableHistoryResultItem;
import org.apache.impala.thrift.TGrantRevokePrivParams;
import org.apache.impala.thrift.TGrantRevokeRoleParams;
+import org.apache.impala.thrift.TImpalaQueryOptions;
import org.apache.impala.thrift.TLineageGraph;
import org.apache.impala.thrift.TLoadDataReq;
import org.apache.impala.thrift.TLoadDataResp;
@@ -257,10 +258,8 @@ public class Frontend {
private static final String VERDICT = "Verdict";
private static final String MEMORY_MAX = "MemoryMax";
private static final String MEMORY_ASK = "MemoryAsk";
- private static final String MEMORY_ASK_UNBOUNDED = "MemoryAskUnbounded";
private static final String CPU_MAX = "CpuMax";
private static final String CPU_ASK = "CpuAsk";
- private static final String CPU_ASK_UNBOUNDED = "CpuAskUnbounded";
/**
* Plan-time context that allows capturing various artifacts created
@@ -2074,8 +2073,6 @@ public class Frontend {
int attempt = 0;
int lastExecutorGroupTotalCores =
expectedTotalCores(executorGroupSetsToUse.get(num_executor_group_sets - 1));
- long memoryAskUnbounded = -1;
- int cpuAskUnbounded = -1;
int i = 0;
while (i < num_executor_group_sets) {
group_set = executorGroupSetsToUse.get(i);
@@ -2175,6 +2172,16 @@ public class Frontend {
int scaled_cores_requirement = -1;
if (isComputeCost) {
Preconditions.checkState(cores_requirement > 0);
+ if (queryOptions.getProcessing_cost_min_threads()
+ > queryOptions.getMax_fragment_instances_per_node()) {
+ throw new AnalysisException(
+ TImpalaQueryOptions.PROCESSING_COST_MIN_THREADS.name() + " ("
+ + queryOptions.getProcessing_cost_min_threads()
+ + ") can not be larger than "
+ + TImpalaQueryOptions.MAX_FRAGMENT_INSTANCES_PER_NODE.name() + " ("
+ + queryOptions.getMax_fragment_instances_per_node() + ").");
+ }
+
scaled_cores_requirement = (int) Math.min(Integer.MAX_VALUE,
Math.ceil(
cores_requirement / BackendConfig.INSTANCE.getQueryCpuCountDivisor()));
@@ -2184,19 +2191,6 @@ public class Frontend {
groupSetProfile, new TCounter(CPU_ASK, TUnit.UNIT, scaled_cores_requirement));
addCounter(groupSetProfile,
new TCounter(EFFECTIVE_PARALLELISM, TUnit.UNIT, cores_requirement));
-
- if (memoryAskUnbounded > 0) {
- addCounter(groupSetProfile,
- new TCounter(MEMORY_ASK_UNBOUNDED, TUnit.BYTES,
- LongMath.saturatedMultiply(
- expectedNumExecutor(group_set), memoryAskUnbounded)));
- memoryAskUnbounded = -1;
- }
- if (cpuAskUnbounded > 0) {
- addCounter(groupSetProfile,
- new TCounter(CPU_ASK_UNBOUNDED, TUnit.UNIT, cpuAskUnbounded));
- cpuAskUnbounded = -1;
- }
}
boolean matchFound = false;
diff --git a/tests/custom_cluster/test_executor_groups.py b/tests/custom_cluster/test_executor_groups.py
index af94e68f0..0ba856597 100644
--- a/tests/custom_cluster/test_executor_groups.py
+++ b/tests/custom_cluster/test_executor_groups.py
@@ -994,10 +994,8 @@ class TestExecutorGroups(CustomClusterTestSuite):
"Memory and cpu limit checking is skipped."),
"EffectiveParallelism: 13", "ExecutorGroupsConsidered: 1"])
- # Test setting REQUEST_POOL and disabling COMPUTE_PROCESSING_COST
- self._set_query_options({
- 'COMPUTE_PROCESSING_COST': 'false',
- 'REQUEST_POOL': 'root.large'})
+ # Test setting REQUEST_POOL=root.large and disabling COMPUTE_PROCESSING_COST
+ self._set_query_options({'COMPUTE_PROCESSING_COST': 'false'})
self._run_query_and_verify_profile(CPU_TEST_QUERY,
["Query Options (set by configuration): REQUEST_POOL=root.large",
"Executor Group: root.large-group",
@@ -1024,26 +1022,13 @@ class TestExecutorGroups(CustomClusterTestSuite):
["Executor Group: root.large-group", "ExecutorGroupsConsidered: 3",
"Verdict: Match", "CpuAsk: 12"])
- # ENABLE_REPLAN=false should force query to run in tiny group, but high scan
- # parallelism will cause it to exceed the admission control slots.
+ # ENABLE_REPLAN=false should force query to run in first group (tiny).
self._set_query_options({'ENABLE_REPLAN': 'false'})
- result = self.execute_query_expect_failure(self.client, CPU_TEST_QUERY)
- status = ("Rejected query from pool root.tiny: number of admission control slots "
- r"needed \(10\) on backend '.*' is greater than total slots available 8. "
- "Reduce mt_dop to less than 8 to ensure that the query can execute.")
- assert re.search(status, str(result))
-
- # ENABLE_REPLAN=false and MAX_FRAGMENT_INSTANCES_PER_NODE=4 should allow query to run
- # in tiny group.
- self._set_query_options({'MAX_FRAGMENT_INSTANCES_PER_NODE': '4'})
- self._run_query_and_verify_profile(CPU_TEST_QUERY,
+ self._run_query_and_verify_profile(TEST_QUERY,
["Executor Group: root.tiny-group", "ExecutorGroupsConsidered: 1",
"Verdict: Assign to first group because query option ENABLE_REPLAN=false"])
-
- # Unset both ENABLE_REPLAN and MAX_FRAGMENT_INSTANCES_PER_NODE
- self._set_query_options({
- 'ENABLE_REPLAN': '',
- 'MAX_FRAGMENT_INSTANCES_PER_NODE': ''})
+ # Unset ENABLE_REPLAN.
+ self._set_query_options({'ENABLE_REPLAN': ''})
# Trivial query should be assigned to tiny group by Frontend.
# Backend may decide to run it in coordinator only.
@@ -1065,33 +1050,31 @@ class TestExecutorGroups(CustomClusterTestSuite):
["Executor Group:"])
# Test combination of PROCESSING_COST_MIN_THREADS and MAX_FRAGMENT_INSTANCES_PER_NODE.
- self._set_query_options({
- 'PROCESSING_COST_MIN_THREADS': '1',
- 'MAX_FRAGMENT_INSTANCES_PER_NODE': '3'})
+ self._set_query_options({'MAX_FRAGMENT_INSTANCES_PER_NODE': '3'})
self._run_query_and_verify_profile(GROUPING_TEST_QUERY,
["Executor Group: root.large-group", "EffectiveParallelism: 9",
"ExecutorGroupsConsidered: 3"])
- self._set_query_options({
- 'MAX_FRAGMENT_INSTANCES_PER_NODE': '4'})
+ self._set_query_options({'MAX_FRAGMENT_INSTANCES_PER_NODE': '4'})
self._run_query_and_verify_profile(GROUPING_TEST_QUERY,
["Executor Group: root.large-group", "EffectiveParallelism: 12",
"ExecutorGroupsConsidered: 3"])
- self._set_query_options({
- 'PROCESSING_COST_MIN_THREADS': '3',
- 'MAX_FRAGMENT_INSTANCES_PER_NODE': '1'})
+ self._set_query_options({'PROCESSING_COST_MIN_THREADS': '2'})
self._run_query_and_verify_profile(GROUPING_TEST_QUERY,
- ["Executor Group: root.large-group", "EffectiveParallelism: 9",
+ ["Executor Group: root.large-group", "EffectiveParallelism: 12",
"ExecutorGroupsConsidered: 3"])
- self._set_query_options({
- 'PROCESSING_COST_MIN_THREADS': '2',
- 'MAX_FRAGMENT_INSTANCES_PER_NODE': '2'})
+ self._set_query_options({'MAX_FRAGMENT_INSTANCES_PER_NODE': '2'})
self._run_query_and_verify_profile(GROUPING_TEST_QUERY,
- ["Executor Group: root.small-group", "EffectiveParallelism: 2",
+ ["Executor Group: root.small-group", "EffectiveParallelism: 4",
"ExecutorGroupsConsidered: 2"])
+ self._set_query_options({'MAX_FRAGMENT_INSTANCES_PER_NODE': '1'})
+ result = self.execute_query_expect_failure(self.client, CPU_TEST_QUERY)
+ status = (r"PROCESSING_COST_MIN_THREADS \(2\) can not be larger than "
+ r"MAX_FRAGMENT_INSTANCES_PER_NODE \(1\).")
+ assert re.search(status, str(result))
# Unset PROCESSING_COST_MIN_THREADS and MAX_FRAGMENT_INSTANCES_PER_NODE.
self._set_query_options({
- 'PROCESSING_COST_MIN_THREADS': '',
- 'MAX_FRAGMENT_INSTANCES_PER_NODE': ''})
+ 'MAX_FRAGMENT_INSTANCES_PER_NODE': '',
+ 'PROCESSING_COST_MIN_THREADS': ''})
# BEGIN testing count queries
# Test optimized count star query with 1824 scan ranges assign to small group.
@@ -1222,7 +1205,7 @@ class TestExecutorGroups(CustomClusterTestSuite):
# Check resource pools on the Web queries site and admission site
self._verify_query_num_for_resource_pool("root.small", 10)
- self._verify_query_num_for_resource_pool("root.tiny", 6)
+ self._verify_query_num_for_resource_pool("root.tiny", 5)
self._verify_query_num_for_resource_pool("root.large", 12)
self._verify_total_admitted_queries("root.small", 11)
self._verify_total_admitted_queries("root.tiny", 8)
[impala] 03/05: IMPALA-12469: Allow short if statements on a single line in clang-format
Posted by mi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
michaelsmith pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 80af98645baf095765c87cd2287cf49fec86fa4a
Author: Peter Rozsa <pr...@cloudera.com>
AuthorDate: Wed Sep 27 08:51:29 2023 +0200
IMPALA-12469: Allow short if statements on a single line in clang-format
This change allows short if statements and blocks to be put on a single
line, clang-format will keep these constructions as-is.
Change-Id: I65d04f00187648fafab89c82cb01ffd9ea17c4bb
Reviewed-on: http://gerrit.cloudera.org:8080/20513
Reviewed-by: Daniel Becker <da...@cloudera.com>
Reviewed-by: Tamas Mate <tm...@apache.org>
Tested-by: Tamas Mate <tm...@apache.org>
---
.clang-format | 2 ++
1 file changed, 2 insertions(+)
diff --git a/.clang-format b/.clang-format
index 054cb3056..aad646bda 100644
--- a/.clang-format
+++ b/.clang-format
@@ -27,6 +27,8 @@ Language: Java
BasedOnStyle: Chromium
AllowShortCaseLabelsOnASingleLine: true
AllowShortFunctionsOnASingleLine: All
+AllowShortIfStatementsOnASingleLine: true
+AllowShortBlocksOnASingleLine: true
AlwaysBreakBeforeMultilineStrings: true
BreakBeforeTernaryOperators: false
ColumnLimit: 90
[impala] 05/05: IMPALA-12464: Add opens for java.util.regex
Posted by mi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
michaelsmith pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit ae14d78c8f2e8366e67aa8c39f0c02c60862905e
Author: Michael Smith <mi...@cloudera.com>
AuthorDate: Mon Sep 25 11:17:27 2023 -0700
IMPALA-12464: Add opens for java.util.regex
Adds --add-opens for java.util.regex with jamm for JDK 17. Java 15 added
hidden classes to define lambdas, and Jamm is unable to access fields of
hidden classes unless add-opens is added for the class.
Change-Id: I9909985fa15ed1b0bc92b04d6ab2645244a0bee3
Reviewed-on: http://gerrit.cloudera.org:8080/20510
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Michael Smith <mi...@cloudera.com>
---
be/src/common/init.cc | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/be/src/common/init.cc b/be/src/common/init.cc
index 3409dde52..300996a35 100644
--- a/be/src/common/init.cc
+++ b/be/src/common/init.cc
@@ -376,9 +376,10 @@ static Status JavaAddOpens(bool useSizeOf) {
}
for (const string& param : {
- // Needed for jamm and ehcache
+ // Needed for jamm and ehcache (jamm needs it to access lambdas)
"--add-opens=java.base/java.lang=ALL-UNNAMED",
"--add-opens=java.base/java.nio=ALL-UNNAMED",
+ "--add-opens=java.base/java.util.regex=ALL-UNNAMED",
"--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"
}) {
val_out << " " << param;