You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2022/10/08 03:43:09 UTC
[GitHub] [hive] ayushtkn opened a new pull request, #3652: HIVE-26614: Fix adding custom jars in Job Classpath.
ayushtkn opened a new pull request, #3652:
URL: https://github.com/apache/hive/pull/3652
### What changes were proposed in this pull request?
Make the Jar path fully qualified, so as to prevent resolution to fs.defaultFs in any application
### Why are the changes needed?
Some applications don't explicitly assume the path to LFS, rather resolve to fs.defaultFs
### Does this PR introduce _any_ user-facing change?
Nopes.
### How was this patch tested?
Build distribution & tried the same command. Doesn't happen in Tests due to https://github.com/apache/hive/blob/9db31108e1a44b2fb0436134fa78eea79418d4ad/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java#L249-L252
May be specific to query engines as well. Tez I suppose alway expects the Path to be in LFS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3652: HIVE-26614: Fix adding custom jars in Job Classpath.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3652:
URL: https://github.com/apache/hive/pull/3652#issuecomment-1272222599
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3652)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3652&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3652&resolved=false&types=BUG) [6 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3652&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3652&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3652&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3652&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3652&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3652&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3652&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3652&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3652&resolved=false&types=CODE_SMELL) [68 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3652&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3652&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3652&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3652: HIVE-26614: Fix adding custom jars in Job Classpath.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3652:
URL: https://github.com/apache/hive/pull/3652#issuecomment-1272301153
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3652)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3652&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3652&resolved=false&types=BUG) [6 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3652&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3652&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3652&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3652&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3652&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3652&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3652&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3652&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3652&resolved=false&types=CODE_SMELL) [68 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3652&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3652&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3652&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #3652: HIVE-26614: Fix adding custom jars in Job Classpath.
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on code in PR #3652:
URL: https://github.com/apache/hive/pull/3652#discussion_r991035653
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:
##########
@@ -4158,7 +4158,7 @@ public static void addDependencyJars(Configuration conf, Class<?>... classes)
if (!localFs.exists(new Path(path))) {
throw new RuntimeException("Could not validate jar file " + path + " for class " + clazz);
}
- jars.add(path);
+ jars.add(localFs.makeQualified(new Path(path)).toString());
Review Comment:
should we use the FileUtils helper method?
````
FileUtils.makeQualified(path, conf).toString()
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #3652: HIVE-26614: Fix adding custom jars in Job Classpath.
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on code in PR #3652:
URL: https://github.com/apache/hive/pull/3652#discussion_r991035653
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:
##########
@@ -4158,7 +4158,7 @@ public static void addDependencyJars(Configuration conf, Class<?>... classes)
if (!localFs.exists(new Path(path))) {
throw new RuntimeException("Could not validate jar file " + path + " for class " + clazz);
}
- jars.add(path);
+ jars.add(localFs.makeQualified(new Path(path)).toString());
Review Comment:
could we use the FileUtils helper method?
````
FileUtils.makeQualified(path, conf).toString()
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ayushtkn commented on a diff in pull request #3652: HIVE-26614: Fix adding custom jars in Job Classpath.
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on code in PR #3652:
URL: https://github.com/apache/hive/pull/3652#discussion_r991105608
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:
##########
@@ -4158,7 +4158,7 @@ public static void addDependencyJars(Configuration conf, Class<?>... classes)
if (!localFs.exists(new Path(path))) {
throw new RuntimeException("Could not validate jar file " + path + " for class " + clazz);
}
- jars.add(path);
+ jars.add(localFs.makeQualified(new Path(path)).toString());
Review Comment:
No, I don't think that will work. It will make it qualified with respect to the FileSystem specified in `Fs.defaultFs` and will lead to the same problem. We need to make it qualified with respect to the `LocalFileSystem`
From the Javadoc
```
* @return path qualified relative to default file system
*/
public static Path makeQualified(Path path, Configuration conf) throws IOException {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ merged pull request #3652: HIVE-26614: Fix adding custom jars in Job Classpath.
Posted by GitBox <gi...@apache.org>.
deniskuzZ merged PR #3652:
URL: https://github.com/apache/hive/pull/3652
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org