You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by mikewalch <gi...@git.apache.org> on 2017/01/30 21:53:18 UTC
[GitHub] accumulo-testing pull request #3: ACCUMULO-4579 Fixed hadoop config bug in a...
GitHub user mikewalch opened a pull request:
https://github.com/apache/accumulo-testing/pull/3
ACCUMULO-4579 Fixed hadoop config bug in accumulo-testing
* TestEnv now returns Hadoop configuration that is loaded from config files
but expects HADOOP_PREFIX to be set in environment
* Fixed bug where Twill was being improperly configured to look in wrong
location for shaded jar when running test application in YARN.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mikewalch/accumulo-testing accumulo-4579
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/accumulo-testing/pull/3.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3
----
commit dd8cad26c82b7dc4f446873b855fe52d5d87046a
Author: Mike Walch <mw...@apache.org>
Date: 2017-01-30T20:03:21Z
ACCUMULO-4579 Fixed hadoop config bug in accumulo-testing
* TestEnv now returns Hadoop configuration that is loaded from config files
but expects HADOOP_PREFIX to be set in environment
* Fixed bug where Twill was being improperly configured to look in wrong
location for shaded jar when running test application in YARN.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo-testing pull request #3: ACCUMULO-4579 Fixed hadoop config bug in a...
Posted by ctubbsii <gi...@git.apache.org>.
Github user ctubbsii commented on a diff in the pull request:
https://github.com/apache/accumulo-testing/pull/3#discussion_r98796795
--- Diff: core/src/main/java/org/apache/accumulo/testing/core/TestEnv.java ---
@@ -96,15 +97,22 @@ public String getPid() {
}
public Configuration getHadoopConfiguration() {
- Configuration config = new Configuration();
- config.set("mapreduce.framework.name", "yarn");
- // Setting below are required due to bundled jar breaking default
- // config.
- // See
- // http://stackoverflow.com/questions/17265002/hadoop-no-filesystem-for-scheme-file
- config.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
- config.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());
- return config;
+ if (hadoopConfig == null) {
+ String hadoopPrefix = System.getenv("HADOOP_PREFIX");
+ if (hadoopPrefix == null || hadoopPrefix.isEmpty()) {
+ throw new IllegalArgumentException("HADOOP_PREFIX must be sent in env");
+ }
+ hadoopConfig = new Configuration();
+ hadoopConfig.addResource(new Path(hadoopPrefix + "/etc/hadoop/core-site.xml"));
--- End diff --
I think using the properties file is better. I don't think we should be writing java code to depend on env variables, especially arbitrary script conventions like `HADOOP_PREFIX`, which is very sensitive to Hadoop packaging/deployment changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo-testing pull request #3: ACCUMULO-4579 Fixed hadoop config bug in a...
Posted by mikewalch <gi...@git.apache.org>.
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/accumulo-testing/pull/3#discussion_r98955888
--- Diff: core/src/main/java/org/apache/accumulo/testing/core/TestEnv.java ---
@@ -96,15 +97,22 @@ public String getPid() {
}
public Configuration getHadoopConfiguration() {
- Configuration config = new Configuration();
- config.set("mapreduce.framework.name", "yarn");
- // Setting below are required due to bundled jar breaking default
- // config.
- // See
- // http://stackoverflow.com/questions/17265002/hadoop-no-filesystem-for-scheme-file
- config.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
- config.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());
- return config;
+ if (hadoopConfig == null) {
+ String hadoopPrefix = System.getenv("HADOOP_PREFIX");
+ if (hadoopPrefix == null || hadoopPrefix.isEmpty()) {
+ throw new IllegalArgumentException("HADOOP_PREFIX must be sent in env");
+ }
+ hadoopConfig = new Configuration();
+ hadoopConfig.addResource(new Path(hadoopPrefix + "/etc/hadoop/core-site.xml"));
--- End diff --
I pushed another commit 62e91527 where I created new properties in `accumulo-testing.properties` to avoid relying on loading Hadoop config files using `HADOOP_PREFIX`. I also added `accumulo` to several properties that configured Accumulo scanners.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo-testing pull request #3: ACCUMULO-4579 Fixed hadoop config bug in a...
Posted by mikewalch <gi...@git.apache.org>.
Github user mikewalch commented on a diff in the pull request:
https://github.com/apache/accumulo-testing/pull/3#discussion_r98556591
--- Diff: core/src/main/java/org/apache/accumulo/testing/core/TestEnv.java ---
@@ -96,15 +97,22 @@ public String getPid() {
}
public Configuration getHadoopConfiguration() {
- Configuration config = new Configuration();
- config.set("mapreduce.framework.name", "yarn");
- // Setting below are required due to bundled jar breaking default
- // config.
- // See
- // http://stackoverflow.com/questions/17265002/hadoop-no-filesystem-for-scheme-file
- config.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
- config.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());
- return config;
+ if (hadoopConfig == null) {
+ String hadoopPrefix = System.getenv("HADOOP_PREFIX");
+ if (hadoopPrefix == null || hadoopPrefix.isEmpty()) {
+ throw new IllegalArgumentException("HADOOP_PREFIX must be sent in env");
+ }
+ hadoopConfig = new Configuration();
+ hadoopConfig.addResource(new Path(hadoopPrefix + "/etc/hadoop/core-site.xml"));
--- End diff --
After talking to @keith-turner offline, I am going to submit an update that pull the needed Hadoop configuration from the accumulo-testing.properties file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo-testing pull request #3: ACCUMULO-4579 Fixed hadoop config bug in a...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/accumulo-testing/pull/3
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---