You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "okumin (via GitHub)" <gi...@apache.org> on 2023/06/11 15:53:24 UTC
[GitHub] [hive] okumin opened a new pull request, #4404: HIVE-24485: Make the slow-start behavior tunable
okumin opened a new pull request, #4404:
URL: https://github.com/apache/hive/pull/4404
### What changes were proposed in this pull request?
Let Hive respect `tez.shuffle-vertex-manager.min-src-fraction` and `tez.shuffle-vertex-manager.max-src-fraction` when auto reducer parallelism is enabled. Currently, we don't have any methods to configure the parameters.
https://issues.apache.org/jira/browse/HIVE-24485
### Why are the changes needed?
We sometimes want to tune the parameters so that Tez can finalize the number of reducers with a more accurate estimation. For example, in extreme cases where the previous stage is very skewed, we may want to configure both params at `1.0`.
### Does this PR introduce _any_ user-facing change?
The behavior will change if a user configures `tez.shuffle-vertex-manager.{min,max}-src-fraction`.
### How was this patch tested?
```
zookage@client-node-0:~$ beeline -e '
> SET hive.tez.auto.reducer.parallelism=true;
> SET hive.tez.min.partition.factor=1.0; -- enforce auto-parallelism
> SET tez.shuffle-vertex-manager.min-src-fraction=0.55;
> SET tez.shuffle-vertex-manager.max-src-fraction=0.95;
> CREATE TABLE mofu (name string);
> INSERT INTO mofu (name) VALUES ('12345');
> SELECT name, count(*) FROM mofu GROUP BY name;
> '
```
```
2023-06-11 15:50:57,287 [INFO] [Dispatcher thread {Central}] |vertexmanager.ShuffleVertexManagerBase|: Settings minFrac: 0.55 maxFrac: 0.95 auto: true desiredTaskIput: 256000000
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] okumin commented on a diff in pull request #4404: HIVE-24485: Make the slow-start behavior tunable
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #4404:
URL: https://github.com/apache/hive/pull/4404#discussion_r1226183179
##########
ql/src/java/org/apache/hadoop/hive/ql/plan/TezEdgeProperty.java:
##########
@@ -51,19 +53,22 @@ public TezEdgeProperty(HiveConf hiveConf, EdgeType edgeType,
}
public TezEdgeProperty(HiveConf hiveConf, EdgeType edgeType, boolean isAutoReduce,
- boolean isSlowStart, int minReducer, int maxReducer, long bytesPerReducer) {
+ boolean isSlowStart, int minReducer, int maxReducer, long bytesPerReducer,
+ float minSrcFraction, float maxSrcFraction) {
this(hiveConf, edgeType, -1);
- setAutoReduce(hiveConf, isAutoReduce, minReducer, maxReducer, bytesPerReducer);
+ setAutoReduce(hiveConf, isAutoReduce, minReducer, maxReducer, bytesPerReducer, minSrcFraction, maxSrcFraction);
this.isSlowStart = isSlowStart;
}
public void setAutoReduce(HiveConf hiveConf, boolean isAutoReduce, int minReducer,
- int maxReducer, long bytesPerReducer) {
+ int maxReducer, long bytesPerReducer, float minSrcFraction, float maxSrcFraction) {
this.hiveConf = hiveConf;
this.minReducer = minReducer;
this.maxReducer = maxReducer;
this.isAutoReduce = isAutoReduce;
this.inputSizePerReducer = bytesPerReducer;
+ this.minSrcFraction = minSrcFraction;
+ this.maxSrcFraction = maxSrcFraction;
Review Comment:
@abstractdog Thanks for putting your opinion. Actually, all the other 3 params are already covered indirectly. The newly added 2 are only ones we can't control at all.
https://github.com/apache/hive/blob/b55f9c513b25e8e61150f6bf0afd1a882780d098/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L1657-L1669
- `tez.shuffle-vertex-manager.enable.auto-parallel`: This is configured when `hive.tez.auto.reducer.parallelism` is enabled and the reduce vertex can be parallelized
- `tez.shuffle-vertex-manager.desired-task-input-size`: This is inherited from `hive.exec.reducers.bytes.per.reducer`
- `tez.shuffle-vertex-manager.min-task-parallelism`: This will be `{estimated # of reducers} * hive.tez.min.partition.factor`
If we support them, it means `tez.shuffle-vertex-manager.*` overwrites the 3 params. Is it preferable?
Another option is to introduce new params like `hive.tez.auto.reducer.parallelism.min-src-fraction` assuming `tez.shuffle-vertex-manager.*` is SPI-like params. It might be consistent with other params.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] okumin commented on a diff in pull request #4404: HIVE-24485: Make the slow-start behavior tunable
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #4404:
URL: https://github.com/apache/hive/pull/4404#discussion_r1225853705
##########
ql/src/java/org/apache/hadoop/hive/ql/plan/TezEdgeProperty.java:
##########
@@ -51,19 +53,22 @@ public TezEdgeProperty(HiveConf hiveConf, EdgeType edgeType,
}
public TezEdgeProperty(HiveConf hiveConf, EdgeType edgeType, boolean isAutoReduce,
- boolean isSlowStart, int minReducer, int maxReducer, long bytesPerReducer) {
+ boolean isSlowStart, int minReducer, int maxReducer, long bytesPerReducer,
+ float minSrcFraction, float maxSrcFraction) {
this(hiveConf, edgeType, -1);
- setAutoReduce(hiveConf, isAutoReduce, minReducer, maxReducer, bytesPerReducer);
+ setAutoReduce(hiveConf, isAutoReduce, minReducer, maxReducer, bytesPerReducer, minSrcFraction, maxSrcFraction);
this.isSlowStart = isSlowStart;
}
public void setAutoReduce(HiveConf hiveConf, boolean isAutoReduce, int minReducer,
- int maxReducer, long bytesPerReducer) {
+ int maxReducer, long bytesPerReducer, float minSrcFraction, float maxSrcFraction) {
this.hiveConf = hiveConf;
this.minReducer = minReducer;
this.maxReducer = maxReducer;
this.isAutoReduce = isAutoReduce;
this.inputSizePerReducer = bytesPerReducer;
+ this.minSrcFraction = minSrcFraction;
+ this.maxSrcFraction = maxSrcFraction;
Review Comment:
I think [the first commit](https://github.com/apache/hive/pull/4404/commits/c87b4ca6652dffc3cc150e54c793477108d8a74f) is the simplest.
However, Gopal suggested I should move to the current style while the previous PR is open.
https://issues.apache.org/jira/browse/HIVE-24485?focusedCommentId=17278522&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17278522
I personally don't have very strong opinions about how to support the params.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a diff in pull request #4404: HIVE-24485: Make the slow-start behavior tunable
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4404:
URL: https://github.com/apache/hive/pull/4404#discussion_r1226352679
##########
ql/src/java/org/apache/hadoop/hive/ql/plan/TezEdgeProperty.java:
##########
@@ -51,19 +53,22 @@ public TezEdgeProperty(HiveConf hiveConf, EdgeType edgeType,
}
public TezEdgeProperty(HiveConf hiveConf, EdgeType edgeType, boolean isAutoReduce,
- boolean isSlowStart, int minReducer, int maxReducer, long bytesPerReducer) {
+ boolean isSlowStart, int minReducer, int maxReducer, long bytesPerReducer,
+ float minSrcFraction, float maxSrcFraction) {
this(hiveConf, edgeType, -1);
- setAutoReduce(hiveConf, isAutoReduce, minReducer, maxReducer, bytesPerReducer);
+ setAutoReduce(hiveConf, isAutoReduce, minReducer, maxReducer, bytesPerReducer, minSrcFraction, maxSrcFraction);
this.isSlowStart = isSlowStart;
}
public void setAutoReduce(HiveConf hiveConf, boolean isAutoReduce, int minReducer,
- int maxReducer, long bytesPerReducer) {
+ int maxReducer, long bytesPerReducer, float minSrcFraction, float maxSrcFraction) {
this.hiveConf = hiveConf;
this.minReducer = minReducer;
this.maxReducer = maxReducer;
this.isAutoReduce = isAutoReduce;
this.inputSizePerReducer = bytesPerReducer;
+ this.minSrcFraction = minSrcFraction;
+ this.maxSrcFraction = maxSrcFraction;
Review Comment:
thanks for the detailed explanation @okumin, I prefer not changing the existing behavior regarding the other properties
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4404: HIVE-24485: Make the slow-start behavior tunable
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4404:
URL: https://github.com/apache/hive/pull/4404#issuecomment-1586247744
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4404)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4404&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4404&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4404&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4404&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4404&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4404&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4404&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4404&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4404&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4404&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4404&resolved=false&types=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4404&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4404&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4404&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4404: HIVE-24485: Make the slow-start behavior tunable
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4404:
URL: https://github.com/apache/hive/pull/4404#issuecomment-1586485712
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4404)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4404&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4404&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4404&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4404&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4404&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4404&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4404&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4404&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4404&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4404&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4404&resolved=false&types=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4404&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4404&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4404&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a diff in pull request #4404: HIVE-24485: Make the slow-start behavior tunable
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4404:
URL: https://github.com/apache/hive/pull/4404#discussion_r1226132855
##########
ql/src/java/org/apache/hadoop/hive/ql/plan/TezEdgeProperty.java:
##########
@@ -51,19 +53,22 @@ public TezEdgeProperty(HiveConf hiveConf, EdgeType edgeType,
}
public TezEdgeProperty(HiveConf hiveConf, EdgeType edgeType, boolean isAutoReduce,
- boolean isSlowStart, int minReducer, int maxReducer, long bytesPerReducer) {
+ boolean isSlowStart, int minReducer, int maxReducer, long bytesPerReducer,
+ float minSrcFraction, float maxSrcFraction) {
this(hiveConf, edgeType, -1);
- setAutoReduce(hiveConf, isAutoReduce, minReducer, maxReducer, bytesPerReducer);
+ setAutoReduce(hiveConf, isAutoReduce, minReducer, maxReducer, bytesPerReducer, minSrcFraction, maxSrcFraction);
this.isSlowStart = isSlowStart;
}
public void setAutoReduce(HiveConf hiveConf, boolean isAutoReduce, int minReducer,
- int maxReducer, long bytesPerReducer) {
+ int maxReducer, long bytesPerReducer, float minSrcFraction, float maxSrcFraction) {
this.hiveConf = hiveConf;
this.minReducer = minReducer;
this.maxReducer = maxReducer;
this.isAutoReduce = isAutoReduce;
this.inputSizePerReducer = bytesPerReducer;
+ this.minSrcFraction = minSrcFraction;
+ this.maxSrcFraction = maxSrcFraction;
Review Comment:
@okumin : thanks for working on this, I believe the latest patch with Gopal's suggestion looks good, and I feel we can support all (currently 5) properties like this:
https://github.com/apache/tez/blob/ccad71dabceb850b58ff38e77fd206e68f62ce4b/tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/ShuffleVertexManager.java#L71-L110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] okumin commented on pull request #4404: HIVE-24485: Make the slow-start behavior tunable
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on PR #4404:
URL: https://github.com/apache/hive/pull/4404#issuecomment-1586944554
Thanks for reviewing this PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog merged pull request #4404: HIVE-24485: Make the slow-start behavior tunable
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog merged PR #4404:
URL: https://github.com/apache/hive/pull/4404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org