You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/04/23 11:12:26 UTC
[GitHub] [hadoop-ozone] maobaolong opened a new pull request #862: HDDS-3482. Impl getUriDefaultPort to BasicOzFsOzFs, return -1 forever to pass the checkPath
maobaolong opened a new pull request #862:
URL: https://github.com/apache/hadoop-ozone/pull/862
## What changes were proposed in this pull request?
For enable MR on yarn with ozone
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3482
## How was this patch tested?
test by submit a mr job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #862: HDDS-3482. Impl getUriDefaultPort to BasicOzFsOzFs, return -1 forever to pass the checkPath
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #862:
URL: https://github.com/apache/hadoop-ozone/pull/862#issuecomment-619518489
@adoroszlai Thank you for help me passed the CI check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #862: HDDS-3482. Impl getUriDefaultPort to BasicOzFsOzFs, return -1 forever to pass the checkPath
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #862:
URL: https://github.com/apache/hadoop-ozone/pull/862#issuecomment-621732469
Merged with d849904c5
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #862: HDDS-3482. Impl getUriDefaultPort to BasicOzFsOzFs, return -1 forever to pass the checkPath
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #862:
URL: https://github.com/apache/hadoop-ozone/pull/862#issuecomment-618755727
Connecting to raw.githubusercontent.com|151.101.52.133|:443... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #862: HDDS-3482. Impl getUriDefaultPort to BasicOzFsOzFs, return -1 forever to pass the checkPath
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #862:
URL: https://github.com/apache/hadoop-ozone/pull/862#issuecomment-620461708
Thanks to post this patch @maobaolong. Can you please give more details how can it be reproduced? We have MR tests for `compose/ozone-mr` clusters which are executed for each PR. I am wondering if we need to modify them as this problem was not caught by them.
Did you use any specific config compared to this one:
https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-config
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #862: HDDS-3482. Impl getUriDefaultPort to BasicOzFsOzFs, return -1 forever to pass the checkPath
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #862:
URL: https://github.com/apache/hadoop-ozone/pull/862#issuecomment-620504150
@elek Thank you for your review.
The https://issues.apache.org/jira/browse/HDDS-3482 given some of my main env.
I use the following suite
- hadoop-2.7.2 as yarn client
- hadoop-2.7.2 as yarn service (RM,NM)
- ozone-0.6.0-SNAPSHOT
So that, the core-site.xml of my yarn client is the following:
Notice: it is different from your `docker-config`, i see you use the higher version hadoop, instead, i use hadoop-2.7.2
```xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>o3fs://mybucket.myvol/</value>
</property>
<property>
<name>fs.o3fs.impl</name>
<value>org.apache.hadoop.fs.ozone.BasicOzoneFileSystem</value>
</property>
<property>
<name>fs.AbstractFileSystem.o3fs.impl</name>
<value>org.apache.hadoop.fs.ozone.BasicOzFs</value>
</property>
</configuration>
```
I executed the blow command to submit the MR to Yarn cluster, but it is failed.
```bash
bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar teragen 1000 /input13
```
The blow is error log after execute the above command.
```
2020-04-23 15:15:04,544 [main] INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/root/.staging/job_1587619434776_0007
org.apache.hadoop.fs.InvalidPathException: Invalid path name Wrong FS: o3fs://mybucket.myvol/tmp/hadoop-yarn/staging/root/.staging/job_1587619434776_0007, expected: o3fs://mybucket.myvol/
at org.apache.hadoop.fs.AbstractFileSystem.checkPath(AbstractFileSystem.java:390)
at org.apache.hadoop.fs.AbstractFileSystem.resolvePath(AbstractFileSystem.java:466)
at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:348)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:285)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
at org.apache.hadoop.examples.terasort.TeraGen.run(TeraGen.java:301)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.terasort.TeraGen.main(TeraGen.java:305)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
```
After some debug work, i focus on the following method from the hadoop-2.7.2.
```java
public void checkPath(Path path) {
URI uri = path.toUri();
String thatScheme = uri.getScheme();
String thatAuthority = uri.getAuthority();
if (thatScheme == null) {
if (thatAuthority == null) {
if (path.isUriPathAbsolute()) {
return;
}
throw new InvalidPathException("relative paths not allowed:" +
path);
} else {
throw new InvalidPathException(
"Path without scheme with non-null authority:" + path);
}
}
String thisScheme = this.getUri().getScheme();
String thisHost = this.getUri().getHost();
String thatHost = uri.getHost();
// Schemes and hosts must match.
// Allow for null Authority for file:///
if (!thisScheme.equalsIgnoreCase(thatScheme) ||
(thisHost != null &&
!thisHost.equalsIgnoreCase(thatHost)) ||
(thisHost == null && thatHost != null)) {
throw new InvalidPathException("Wrong FS: " + path + ", expected: "
+ this.getUri());
}
// Ports must match, unless this FS instance is using the default port, in
// which case the port may be omitted from the given URI
int thisPort = this.getUri().getPort();
int thatPort = uri.getPort();
if (thatPort == -1) { // -1 => defaultPort of Uri scheme
thatPort = this.getUriDefaultPort();
}
if (thisPort != thatPort) {
throw new InvalidPathException("Wrong FS: " + path + ", expected: "
+ this.getUri());
}
}
```
`thisPort` is -1, and `thatPort` is -1 start, but `thatPort` is assigned by the abstract method `getUriDefaultPort()`, it is impl by DelegateToFileSystem which ozfs and OzfsBasic extended.
unfortunately, this method impl good before we merge the patch https://github.com/apache/hadoop/commit/dc065dd64ca3e101b0c0a7bcc7d7a067b77d6c82
And key point is, we are using 2.7.2, so the `getUriDefaultPort` return 0. I have to override this method in the `BasicOzFs`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #862: HDDS-3482. Impl getUriDefaultPort to BasicOzFsOzFs, return -1 forever to pass the checkPath
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #862:
URL: https://github.com/apache/hadoop-ozone/pull/862#issuecomment-621716069
Thanks to explain it. Sorry, I missed the description in the Jira, (side note: I like your detailed problem descriptions).
Based on this description we don't need to fix our acceptance tests as it uses latest 2.7.x (which includes HADOOP-12991) but good to have this patch to support older Hadoop 2.7.2
Thank you very much for the fix and the detailed analyses @maobaolong
I am merging it now.
Other side note: In HDDS-3458 I am planning to refactor this part to remove custom class-loader magic with a simplified build-time solution...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on issue #862: HDDS-3482. Impl getUriDefaultPort to BasicOzFsOzFs, return -1 forever to pass the checkPath
Posted by GitBox <gi...@apache.org>.
maobaolong commented on issue #862:
URL: https://github.com/apache/hadoop-ozone/pull/862#issuecomment-618391539
The failed test seems is not caused by me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek removed a comment on pull request #862: HDDS-3482. Impl getUriDefaultPort to BasicOzFsOzFs, return -1 forever to pass the checkPath
Posted by GitBox <gi...@apache.org>.
elek removed a comment on pull request #862:
URL: https://github.com/apache/hadoop-ozone/pull/862#issuecomment-621716069
Thanks to explain it. Sorry, I missed the description in the Jira, (side note: I like your detailed problem descriptions).
Based on this description we don't need to fix our acceptance tests as it uses latest 2.7.x (which includes HADOOP-12991) but good to have this patch to support older Hadoop 2.7.2
Thank you very much for the fix and the detailed analyses @maobaolong
I am merging it now.
Other side note: In HDDS-3458 I am planning to refactor this part to remove custom class-loader magic with a simplified build-time solution...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org