You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "ngsg (via GitHub)" <gi...@apache.org> on 2025/03/28 12:52:29 UTC
Re: [PR] HIVE-28548: Subdivide memory size allocated to parallel operators [hive]
ngsg commented on code in PR #5478:
URL: https://github.com/apache/hive/pull/5478#discussion_r2018602715
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java:
##########
@@ -389,14 +390,14 @@ protected void initializeOp(Configuration hconf) throws HiveException {
isLlap = LlapDaemonInfo.INSTANCE.isLlap();
numExecutors = isLlap ? LlapDaemonInfo.INSTANCE.getNumExecutors() : 1;
firstRow = true;
+ memoryMXBean = ManagementFactory.getMemoryMXBean();
+ maxMemory = isTez ? getConf().getMaxMemoryAvailable() : memoryMXBean.getHeapMemoryUsage().getMax();
Review Comment:
I think we could use a number less than `operators.size()` since not all operators require significant memory that should be managed by `getMaxMemoryAvailable()`. For instance, we can likely ignore select and filter operators, as they behave more like fire-and-forget operations and do not consume much memory. Given that `OperatorDesc.getMaxMemoryAvailable()` is referred by MapJoin, ReduceSink (with TopN), and GroupBy (with Hash Aggregate Mode), I think #MapJoin + #(RS with TopN) + #(GBY with HashMode) + 1(for the rest) could be a candidate.
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java:
##########
@@ -389,14 +390,14 @@ protected void initializeOp(Configuration hconf) throws HiveException {
isLlap = LlapDaemonInfo.INSTANCE.isLlap();
numExecutors = isLlap ? LlapDaemonInfo.INSTANCE.getNumExecutors() : 1;
firstRow = true;
+ memoryMXBean = ManagementFactory.getMemoryMXBean();
+ maxMemory = isTez ? getConf().getMaxMemoryAvailable() : memoryMXBean.getHeapMemoryUsage().getMax();
Review Comment:
I think we could use a number less than `operators.size()` since not all operators require significant memory that should be managed by `getMaxMemoryAvailable()`. For instance, we can likely ignore select and filter operators, as they behave more like fire-and-forget operations and do not consume much memory. Given that `OperatorDesc.getMaxMemoryAvailable()` is referred by MapJoin, ReduceSink (with TopN), and GroupBy (with Hash Aggregate Mode), I think #MapJoin + #(RS with TopN) + #(GBY with HashMode) + 1(for the rest) could be a candidate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org