You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/01/14 21:59:27 UTC
[GitHub] [hive] johan-g opened a new pull request #1870: HIVE-24637 make Tez progress log interval configurable
johan-g opened a new pull request #1870:
URL: https://github.com/apache/hive/pull/1870
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
This makes the Hive on tez progress log interval configurable using the same parameter used in map reduce
### Why are the changes needed?
For long running Hive jobs the current hardcoded value of 3seconds can result in very large logs
### Does this PR introduce _any_ user-facing change?
Default interval of logging progress log when running on Tez changes from 3 to 6 sec
### How was this patch tested?
Internal Hive deployment
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk commented on a change in pull request #1870: HIVE-24637 make Tez progress log interval configurable
Posted by GitBox <gi...@apache.org>.
kgyrtkirk commented on a change in pull request #1870:
URL: https://github.com/apache/hive/pull/1870#discussion_r561895307
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java
##########
@@ -39,7 +40,16 @@
}
private abstract static class BaseUpdateFunction implements UpdateFunction {
- private static final int PRINT_INTERVAL = 3000;
+ private static final long PRINT_INTERVAL;
+
+ static {
Review comment:
I think instead of initializing this from a vanilla `new HiveConf` in a static block - we should access this setting from the constructor thru the `monitor` which seem to have a `hiveconf` field
cc: @abstractdog
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk commented on a change in pull request #1870: HIVE-24637 make Tez progress log interval configurable
Posted by GitBox <gi...@apache.org>.
kgyrtkirk commented on a change in pull request #1870:
URL: https://github.com/apache/hive/pull/1870#discussion_r563673300
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java
##########
@@ -39,7 +40,16 @@
}
private abstract static class BaseUpdateFunction implements UpdateFunction {
- private static final int PRINT_INTERVAL = 3000;
+ private static final long PRINT_INTERVAL;
+
+ static {
Review comment:
This approach seems to be great!
Yes you could get it from SS as well ; but in some cases there might be no seesion in some cases - which should be handled differently.
Thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] johan-g commented on a change in pull request #1870: HIVE-24637 make Tez progress log interval configurable
Posted by GitBox <gi...@apache.org>.
johan-g commented on a change in pull request #1870:
URL: https://github.com/apache/hive/pull/1870#discussion_r562102132
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java
##########
@@ -39,7 +40,16 @@
}
private abstract static class BaseUpdateFunction implements UpdateFunction {
- private static final int PRINT_INTERVAL = 3000;
+ private static final long PRINT_INTERVAL;
+
+ static {
Review comment:
Thanks for the feedback @kgyrtkirk, I've added a `getHiveConf` function to `TezJobMonitor` and move this logic to the constructor of `BaseUpdateFunction`. I guess another approach could be to get `HiveConf` from `SessionState` which is used in the constructor for `perfLogger` already, but grepping for other cases of this approach it doesn't seem to common.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] johan-g commented on a change in pull request #1870: HIVE-24637 make Tez progress log interval configurable
Posted by GitBox <gi...@apache.org>.
johan-g commented on a change in pull request #1870:
URL: https://github.com/apache/hive/pull/1870#discussion_r563922022
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java
##########
@@ -39,7 +40,16 @@
}
private abstract static class BaseUpdateFunction implements UpdateFunction {
- private static final int PRINT_INTERVAL = 3000;
+ private static final long PRINT_INTERVAL;
+
+ static {
Review comment:
Thanks for the feedback 🙇
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk commented on a change in pull request #1870: HIVE-24637 make Tez progress log interval configurable
Posted by GitBox <gi...@apache.org>.
kgyrtkirk commented on a change in pull request #1870:
URL: https://github.com/apache/hive/pull/1870#discussion_r561895307
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java
##########
@@ -39,7 +40,16 @@
}
private abstract static class BaseUpdateFunction implements UpdateFunction {
- private static final int PRINT_INTERVAL = 3000;
+ private static final long PRINT_INTERVAL;
+
+ static {
Review comment:
I think instead of initializing this from a vanilla `new HiveConf` in a static block - we should access this setting from the constructor thru the `monitor` which seem to have a `hiveconf` field
cc: @abstractdog
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] johan-g commented on a change in pull request #1870: HIVE-24637 make Tez progress log interval configurable
Posted by GitBox <gi...@apache.org>.
johan-g commented on a change in pull request #1870:
URL: https://github.com/apache/hive/pull/1870#discussion_r563922022
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java
##########
@@ -39,7 +40,16 @@
}
private abstract static class BaseUpdateFunction implements UpdateFunction {
- private static final int PRINT_INTERVAL = 3000;
+ private static final long PRINT_INTERVAL;
+
+ static {
Review comment:
Thanks for the feedback 🙇
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk merged pull request #1870: HIVE-24637 make Tez progress log interval configurable
Posted by GitBox <gi...@apache.org>.
kgyrtkirk merged pull request #1870:
URL: https://github.com/apache/hive/pull/1870
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk commented on a change in pull request #1870: HIVE-24637 make Tez progress log interval configurable
Posted by GitBox <gi...@apache.org>.
kgyrtkirk commented on a change in pull request #1870:
URL: https://github.com/apache/hive/pull/1870#discussion_r563673300
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java
##########
@@ -39,7 +40,16 @@
}
private abstract static class BaseUpdateFunction implements UpdateFunction {
- private static final int PRINT_INTERVAL = 3000;
+ private static final long PRINT_INTERVAL;
+
+ static {
Review comment:
This approach seems to be great!
Yes you could get it from SS as well ; but in some cases there might be no seesion in some cases - which should be handled differently.
Thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] johan-g commented on a change in pull request #1870: HIVE-24637 make Tez progress log interval configurable
Posted by GitBox <gi...@apache.org>.
johan-g commented on a change in pull request #1870:
URL: https://github.com/apache/hive/pull/1870#discussion_r562102132
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java
##########
@@ -39,7 +40,16 @@
}
private abstract static class BaseUpdateFunction implements UpdateFunction {
- private static final int PRINT_INTERVAL = 3000;
+ private static final long PRINT_INTERVAL;
+
+ static {
Review comment:
Thanks for the feedback @kgyrtkirk, I've added a `getHiveConf` function to `TezJobMonitor` and move this logic to the constructor of `BaseUpdateFunction`. I guess another approach could be to get `HiveConf` from `SessionState` which is used in the constructor for `perfLogger` already, but grepping for other cases of this approach it doesn't seem to common.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org