You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/02/03 16:47:58 UTC
[GitHub] [hive] abstractdog opened a new pull request #1941: HIVE-24730: Shims classes override values from hive-site.xml and tez-site.xml silently
abstractdog opened a new pull request #1941:
URL: https://github.com/apache/hive/pull/1941
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] mustafaiman edited a comment on pull request #1941: HIVE-24730: Shims classes override values from hive-site.xml and tez-site.xml silently
Posted by GitBox <gi...@apache.org>.
mustafaiman edited a comment on pull request #1941:
URL: https://github.com/apache/hive/pull/1941#issuecomment-788125172
@abstractdog I still stand with what I wrote on the Jira. I do not understand why we have `overrideIntIfDefaultIsSet(conf, x, y, z);` in the code instead of putting `x=z` in hive-site.xml/tez-site.xml. We do not need any of this override logic. Why don't we just remove all the overriding madness and put the correct values in xml files? What am I missing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #1941: HIVE-24730: Shims classes override values from hive-site.xml and tez-site.xml silently
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #1941:
URL: https://github.com/apache/hive/pull/1941#discussion_r570026024
##########
File path: shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -338,6 +338,24 @@ public void setupConfiguration(Configuration conf) {
conf.setInt(MRJobConfig.REDUCE_MEMORY_MB, 512);
conf.setInt(MRJobConfig.MR_AM_VMEM_MB, 128);
}
+
+ // we want this shim to override these values only if the default value is set, otherwise there
+ // is a good chance that the user of the minicluster set another value intentionally
+ protected void overrideIntIfDefaultIsSet(Configuration conf, String key, int defaultVal,
+ int newVal) {
+ if (conf.getInt(key, defaultVal) == defaultVal) {
+ LOG.info("Hadoop23Shims overrides '{}' from {} to {}", key, defaultVal, newVal);
+ conf.setInt(key, newVal);
+ }
Review comment:
Would it make sense to LOG even if we do not override?
It took me several attempts to understand your comments in the description (I have kept looking for 256 in the logs), so either it is too early morning for me or some additional log messages would be good 😄
If I understand correctly this is test code only change so it would not be a problem if we add more log.
Your thoughts? And BTW thanks for the patch!
Peter
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on pull request #1941: HIVE-24730: Shims classes override values from hive-site.xml and tez-site.xml silently
Posted by GitBox <gi...@apache.org>.
abstractdog commented on pull request #1941:
URL: https://github.com/apache/hive/pull/1941#issuecomment-788132782
> @abstractdog I still stand with what I wrote on the Jira. I do not understand why we have `overrideIntIfDefaultIsSet(conf, x, y, z);` in the code instead of putting `x=z` in hive-site.xml/tez-site.xml. We do not need any of this override logic. Why don't we just remove all the overriding madness and put the correct values in xml files? What am I missing?
okay, I agree, honestly, I created the optional override logic because I assumed that these mini clusters are used somewhere else where someone relies on the fact that mini clusters can ensure proper settings (hive.tez.container.size != -1)...but if so, the above points you mentioned apply to those as well, so they should take care of the proper settings in XMLs or whatever before invoking minicluster initialization
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on pull request #1941: HIVE-24730: Shims classes override values from hive-site.xml and tez-site.xml silently
Posted by GitBox <gi...@apache.org>.
abstractdog commented on pull request #1941:
URL: https://github.com/apache/hive/pull/1941#issuecomment-836749531
merged to master, thanks for the reviews @pvary and @mustafaiman
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] mustafaiman commented on pull request #1941: HIVE-24730: Shims classes override values from hive-site.xml and tez-site.xml silently
Posted by GitBox <gi...@apache.org>.
mustafaiman commented on pull request #1941:
URL: https://github.com/apache/hive/pull/1941#issuecomment-788125172
@abstractdog I still stand with what I wrote on the Jira. I do not understand why we have `overrideIntIfDefaultIsSet(conf, x, y, 128);` in the code instead of putting `x=y` in hive-site.xml/tez-site.xml. We do not need any of this override logic. Why don't we just remove all the overriding madness and put the correct values in xml files? What am I missing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog edited a comment on pull request #1941: HIVE-24730: Shims classes override values from hive-site.xml and tez-site.xml silently
Posted by GitBox <gi...@apache.org>.
abstractdog edited a comment on pull request #1941:
URL: https://github.com/apache/hive/pull/1941#issuecomment-788132782
> @abstractdog I still stand with what I wrote on the Jira. I do not understand why we have `overrideIntIfDefaultIsSet(conf, x, y, z);` in the code instead of putting `x=z` in hive-site.xml/tez-site.xml. We do not need any of this override logic. Why don't we just remove all the overriding madness and put the correct values in xml files? What am I missing?
okay, I agree, honestly, I created the optional override logic because I assumed that these mini clusters are used somewhere else where someone relies on the fact that mini clusters can ensure proper settings (hive.tez.container.size != -1)...but if so, the above points you mentioned apply to those as well, so they should take care of the proper settings in XMLs or whatever before invoking minicluster initialization
I'm about to refresh the patch by completely removing override logic
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a change in pull request #1941: HIVE-24730: Shims classes override values from hive-site.xml and tez-site.xml silently
Posted by GitBox <gi...@apache.org>.
abstractdog commented on a change in pull request #1941:
URL: https://github.com/apache/hive/pull/1941#discussion_r570366496
##########
File path: shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -446,16 +478,23 @@ public void shutdown() throws IOException {
@Override
public void setupConfiguration(Configuration conf) {
Configuration config = mr.getConfig();
- for (Map.Entry<String, String> pair: config) {
+ for (Map.Entry<String, String> pair : config) {
Review comment:
not, but it's autoformatted by eclipse, actually I liked the change, I always put spaces before and after colon in iterations
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #1941: HIVE-24730: Shims classes override values from hive-site.xml and tez-site.xml silently
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #1941:
URL: https://github.com/apache/hive/pull/1941#discussion_r570100150
##########
File path: shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -446,16 +478,23 @@ public void shutdown() throws IOException {
@Override
public void setupConfiguration(Configuration conf) {
Configuration config = mr.getConfig();
- for (Map.Entry<String, String> pair: config) {
+ for (Map.Entry<String, String> pair : config) {
Review comment:
was this intentional?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a change in pull request #1941: HIVE-24730: Shims classes override values from hive-site.xml and tez-site.xml silently
Posted by GitBox <gi...@apache.org>.
abstractdog commented on a change in pull request #1941:
URL: https://github.com/apache/hive/pull/1941#discussion_r570059016
##########
File path: shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -338,6 +338,24 @@ public void setupConfiguration(Configuration conf) {
conf.setInt(MRJobConfig.REDUCE_MEMORY_MB, 512);
conf.setInt(MRJobConfig.MR_AM_VMEM_MB, 128);
}
+
+ // we want this shim to override these values only if the default value is set, otherwise there
+ // is a good chance that the user of the minicluster set another value intentionally
+ protected void overrideIntIfDefaultIsSet(Configuration conf, String key, int defaultVal,
+ int newVal) {
+ if (conf.getInt(key, defaultVal) == defaultVal) {
+ LOG.info("Hadoop23Shims overrides '{}' from {} to {}", key, defaultVal, newVal);
+ conf.setInt(key, newVal);
+ }
Review comment:
added in second commit: https://github.com/apache/hive/pull/1941/commits/697dc4a9e37c12b4d7d34589cc1014d6e08a6700
edited PR description with new log contents and some instructions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #1941: HIVE-24730: Shims classes override values from hive-site.xml and tez-site.xml silently
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #1941:
URL: https://github.com/apache/hive/pull/1941#discussion_r570026024
##########
File path: shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -338,6 +338,24 @@ public void setupConfiguration(Configuration conf) {
conf.setInt(MRJobConfig.REDUCE_MEMORY_MB, 512);
conf.setInt(MRJobConfig.MR_AM_VMEM_MB, 128);
}
+
+ // we want this shim to override these values only if the default value is set, otherwise there
+ // is a good chance that the user of the minicluster set another value intentionally
+ protected void overrideIntIfDefaultIsSet(Configuration conf, String key, int defaultVal,
+ int newVal) {
+ if (conf.getInt(key, defaultVal) == defaultVal) {
+ LOG.info("Hadoop23Shims overrides '{}' from {} to {}", key, defaultVal, newVal);
+ conf.setInt(key, newVal);
+ }
Review comment:
Would it make sense to LOG even if we do not override?
It took me several attempts to understand your comments in the description (I have kept looking for 256 in the logs), so either it is too early morning for me or some additional log messages would be good 😄
If I understand correctly this is test code only change so it would not be a problem if we add more log.
Your thoughts? And BTW thanks for the patch!
Peter
##########
File path: shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -446,16 +478,23 @@ public void shutdown() throws IOException {
@Override
public void setupConfiguration(Configuration conf) {
Configuration config = mr.getConfig();
- for (Map.Entry<String, String> pair: config) {
+ for (Map.Entry<String, String> pair : config) {
Review comment:
was this intentional?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a change in pull request #1941: HIVE-24730: Shims classes override values from hive-site.xml and tez-site.xml silently
Posted by GitBox <gi...@apache.org>.
abstractdog commented on a change in pull request #1941:
URL: https://github.com/apache/hive/pull/1941#discussion_r570366496
##########
File path: shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -446,16 +478,23 @@ public void shutdown() throws IOException {
@Override
public void setupConfiguration(Configuration conf) {
Configuration config = mr.getConfig();
- for (Map.Entry<String, String> pair: config) {
+ for (Map.Entry<String, String> pair : config) {
Review comment:
not, but it's autoformatted by eclipse, actually I liked the change, I always put spaces before and after colon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog merged pull request #1941: HIVE-24730: Shims classes override values from hive-site.xml and tez-site.xml silently
Posted by GitBox <gi...@apache.org>.
abstractdog merged pull request #1941:
URL: https://github.com/apache/hive/pull/1941
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on pull request #1941: HIVE-24730: Shims classes override values from hive-site.xml and tez-site.xml silently
Posted by GitBox <gi...@apache.org>.
pvary commented on pull request #1941:
URL: https://github.com/apache/hive/pull/1941#issuecomment-836527470
+1 pending tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on pull request #1941: HIVE-24730: Shims classes override values from hive-site.xml and tez-site.xml silently
Posted by GitBox <gi...@apache.org>.
abstractdog commented on pull request #1941:
URL: https://github.com/apache/hive/pull/1941#issuecomment-787704534
@mustafaiman : you had some comments on jira, could you please take a look at the commits if you're fine with those? if so, let's merge this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a change in pull request #1941: HIVE-24730: Shims classes override values from hive-site.xml and tez-site.xml silently
Posted by GitBox <gi...@apache.org>.
abstractdog commented on a change in pull request #1941:
URL: https://github.com/apache/hive/pull/1941#discussion_r570030445
##########
File path: shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -338,6 +338,24 @@ public void setupConfiguration(Configuration conf) {
conf.setInt(MRJobConfig.REDUCE_MEMORY_MB, 512);
conf.setInt(MRJobConfig.MR_AM_VMEM_MB, 128);
}
+
+ // we want this shim to override these values only if the default value is set, otherwise there
+ // is a good chance that the user of the minicluster set another value intentionally
+ protected void overrideIntIfDefaultIsSet(Configuration conf, String key, int defaultVal,
+ int newVal) {
+ if (conf.getInt(key, defaultVal) == defaultVal) {
+ LOG.info("Hadoop23Shims overrides '{}' from {} to {}", key, defaultVal, newVal);
+ conf.setInt(key, newVal);
+ }
Review comment:
right, I'm adding logs for untouched configs too
##########
File path: shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -338,6 +338,24 @@ public void setupConfiguration(Configuration conf) {
conf.setInt(MRJobConfig.REDUCE_MEMORY_MB, 512);
conf.setInt(MRJobConfig.MR_AM_VMEM_MB, 128);
}
+
+ // we want this shim to override these values only if the default value is set, otherwise there
+ // is a good chance that the user of the minicluster set another value intentionally
+ protected void overrideIntIfDefaultIsSet(Configuration conf, String key, int defaultVal,
+ int newVal) {
+ if (conf.getInt(key, defaultVal) == defaultVal) {
+ LOG.info("Hadoop23Shims overrides '{}' from {} to {}", key, defaultVal, newVal);
+ conf.setInt(key, newVal);
+ }
Review comment:
added in second commit: https://github.com/apache/hive/pull/1941/commits/697dc4a9e37c12b4d7d34589cc1014d6e08a6700
edited PR description with new log contents and some instructions
##########
File path: shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -446,16 +478,23 @@ public void shutdown() throws IOException {
@Override
public void setupConfiguration(Configuration conf) {
Configuration config = mr.getConfig();
- for (Map.Entry<String, String> pair: config) {
+ for (Map.Entry<String, String> pair : config) {
Review comment:
not, but it's autoformatted by eclipse, actually I liked the change, I always put spaces before and after colon
##########
File path: shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -446,16 +478,23 @@ public void shutdown() throws IOException {
@Override
public void setupConfiguration(Configuration conf) {
Configuration config = mr.getConfig();
- for (Map.Entry<String, String> pair: config) {
+ for (Map.Entry<String, String> pair : config) {
Review comment:
not, but it's autoformatted by eclipse, actually I liked the change, I always put spaces before and after colon in iterations
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a change in pull request #1941: HIVE-24730: Shims classes override values from hive-site.xml and tez-site.xml silently
Posted by GitBox <gi...@apache.org>.
abstractdog commented on a change in pull request #1941:
URL: https://github.com/apache/hive/pull/1941#discussion_r570030445
##########
File path: shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -338,6 +338,24 @@ public void setupConfiguration(Configuration conf) {
conf.setInt(MRJobConfig.REDUCE_MEMORY_MB, 512);
conf.setInt(MRJobConfig.MR_AM_VMEM_MB, 128);
}
+
+ // we want this shim to override these values only if the default value is set, otherwise there
+ // is a good chance that the user of the minicluster set another value intentionally
+ protected void overrideIntIfDefaultIsSet(Configuration conf, String key, int defaultVal,
+ int newVal) {
+ if (conf.getInt(key, defaultVal) == defaultVal) {
+ LOG.info("Hadoop23Shims overrides '{}' from {} to {}", key, defaultVal, newVal);
+ conf.setInt(key, newVal);
+ }
Review comment:
right, I'm adding logs for untouched configs too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org