You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/03/16 17:52:23 UTC
[GitHub] [hive] guptanikhil007 commented on a change in pull request #2065: [WIP] HIVE-24201-WorkloadManager kills query being moved to different pool if destination pool does not have enough sessions
guptanikhil007 commented on a change in pull request #2065:
URL: https://github.com/apache/hive/pull/2065#discussion_r594076379
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
##########
@@ -657,6 +673,30 @@ private void processCurrentEvents(EventState e, WmThreadSyncWork syncWork) throw
}
e.toReuse.clear();
+ // For pools which have queued requests, move the "delayed moves" now
Review comment:
Comment meaning is not clear
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
##########
@@ -794,17 +834,18 @@ private void handleMoveSessionOnMasterThread(final MoveSession moveSession,
final WmThreadSyncWork syncWork,
final HashSet<String> poolsToRedistribute,
final Map<WmTezSession, GetRequest> toReuse,
- final Map<WmTezSession, WmEvent> recordMoveEvents) {
+ final Map<WmTezSession, WmEvent> recordMoveEvents,
+ final boolean moveImmediately) {
Review comment:
nit: indentation fix
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
##########
@@ -816,18 +857,36 @@ private void handleMoveSessionOnMasterThread(final MoveSession moveSession,
LOG.error("Failed to move session: {}. Session is not added to destination.", moveSession);
}
} else {
- WmTezSession session = moveSession.srcSession;
- KillQueryContext killQueryContext = new KillQueryContext(session, "Destination pool " + destPoolName +
- " is full. Killing query.");
- resetAndQueueKill(syncWork.toKillQuery, killQueryContext, toReuse);
+ LOG.error("Failed to move session: {}. Session is not removed from its pool.", moveSession);
}
} else {
- LOG.error("Failed to move session: {}. Session is not removed from its pool.", moveSession);
+ // If delayed move is set to true, don't kill the query. Let the query run in source pool
+ // add the session to the source pool's delayed move sessions so that the session can be moved
+ // later when there is a new request on the source pool.
+ if(HiveConf.getBoolVar(conf, ConfVars.HIVE_SERVER2_WM_DELAYED_MOVE) && !moveImmediately) {
+ LOG.info("Move: {} is a delayed move.Since destination pool {} is full, running in source pool "
+ + "as long as possible.", moveSession, destPoolName);
+ String srcPoolName = moveSession.srcSession.getPoolName();
+ if (srcPoolName != null) {
+ PoolState srcPool = pools.get(srcPoolName);
+ if (srcPool != null) {
+ if (srcPool.delayedMoveSessionsSet.add(moveSession)) {
+ srcPool.delayedMoveSessionsQueue.addLast(moveSession);
Review comment:
We can try using TreeSet instead of a queue and a HashSet
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
##########
@@ -657,6 +673,30 @@ private void processCurrentEvents(EventState e, WmThreadSyncWork syncWork) throw
}
e.toReuse.clear();
+ // For pools which have queued requests, move the "delayed moves" now
Review comment:
Put the code behind the config
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org