You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Silvan Kaiser <si...@quobyte.com> on 2015/05/08 15:52:49 UTC
URI missing scheme and authority in job start with new FileSystem implementation
Hello!
I am running a new FileSystem implementation, testing it with a simple
standalone operation from the Hadoop "Single Node Setup" page. Staging
example jobs works alright but upon scheduling the container management
fails because of an URI without scheme (and authority).
NodeManager log snippet:
...
2015-05-08 15:18:15,331 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Got exception parsing jobSubmitDir/job.splitmetainfo and value resource {
port: -1 file:
"/tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo"
} size: 771 timestamp: 1431091092000 type: FILE visibility: APPLICATION
2015-05-08 15:18:15,331 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Failed to parse resource-request
java.net.URISyntaxException: Expected scheme name at index 0:
:///tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo
...
The path element of the URI is correct, it simply is missing it's scheme
and authority but i cannot find out why so far.
Please note that the URIs used during staging are correct and contain
scheme & authority correctly. This means the incomplete URI is created
during job creation.
My guess so far: this is either because I am missing some configuration for
the Nodemanager (therefore he does not know about my FileSystem
implementation and throws the unknown scheme & authority out the window) or
that the FileSystem implementation has an error / did not override a method
that requires adoption.
Can someone please give me a hint where the URIs used by the NodeManager
are constructed prior to scheduling? This would allow me to trace back
where the incomplete URI is coming from...
Best regards
Silvan
--
Quobyte GmbH
Boyenstr. 41 - 10115 Berlin-Mitte - Germany
+49-30-814 591 800 - www.quobyte.com<http://www.quobyte.com/>
Amtsgericht Berlin-Charlottenburg, HRB 149012B
Management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
--
--
*Quobyte* GmbH
Hardenbergplatz 2 - 10623 Berlin - Germany
+49-30-814 591 800 - www.quobyte.com
Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
Re: URI missing scheme and authority in job start with new FileSystem implementation
Posted by Silvan Kaiser <si...@quobyte.com>.
Hi Varun, hi List!
Just a small success feedback note:
It took me quite a while but in the end i found out that not mine but
AbstractFileSystem.java's resolvePath() method was used, sigh.
Solution was simply to add an override in the DelegateToFileSystem impl,
this override explicitely calls fsImpl.resolvePath() (which is my
FileSytems implementation) and yarn is now writing correct fully qualified
URIs into the job configuration.
Thanks for helping!
Silvan
--
--
*Quobyte* GmbH
Hardenbergplatz 2 - 10623 Berlin - Germany
+49-30-814 591 800 - www.quobyte.com
Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
Re: URI missing scheme and authority in job start with new FileSystem implementation
Posted by Silvan Kaiser <si...@quobyte.com>.
Hi Varun, hi List!
Just a small success feedback note:
It took me quite a while but in the end i found out that not mine but
AbstractFileSystem.java's resolvePath() method was used, sigh.
Solution was simply to add an override in the DelegateToFileSystem impl,
this override explicitely calls fsImpl.resolvePath() (which is my
FileSytems implementation) and yarn is now writing correct fully qualified
URIs into the job configuration.
Thanks for helping!
Silvan
--
--
*Quobyte* GmbH
Hardenbergplatz 2 - 10623 Berlin - Germany
+49-30-814 591 800 - www.quobyte.com
Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
Re: URI missing scheme and authority in job start with new FileSystem implementation
Posted by Silvan Kaiser <si...@quobyte.com>.
Hi Varun, hi List!
Just a small success feedback note:
It took me quite a while but in the end i found out that not mine but
AbstractFileSystem.java's resolvePath() method was used, sigh.
Solution was simply to add an override in the DelegateToFileSystem impl,
this override explicitely calls fsImpl.resolvePath() (which is my
FileSytems implementation) and yarn is now writing correct fully qualified
URIs into the job configuration.
Thanks for helping!
Silvan
--
--
*Quobyte* GmbH
Hardenbergplatz 2 - 10623 Berlin - Germany
+49-30-814 591 800 - www.quobyte.com
Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
Re: URI missing scheme and authority in job start with new FileSystem implementation
Posted by Silvan Kaiser <si...@quobyte.com>.
Hi Varun, hi List!
Just a small success feedback note:
It took me quite a while but in the end i found out that not mine but
AbstractFileSystem.java's resolvePath() method was used, sigh.
Solution was simply to add an override in the DelegateToFileSystem impl,
this override explicitely calls fsImpl.resolvePath() (which is my
FileSytems implementation) and yarn is now writing correct fully qualified
URIs into the job configuration.
Thanks for helping!
Silvan
--
--
*Quobyte* GmbH
Hardenbergplatz 2 - 10623 Berlin - Germany
+49-30-814 591 800 - www.quobyte.com
Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
Re: URI missing scheme and authority in job start with new FileSystem implementation
Posted by Silvan Kaiser <si...@quobyte.com>.
Dear Varun,
thanks for the hints! Yep it's Mapreduce.
Can't look at the code right now but I cannot remember overriding resolvePath. Will check it out and give feedback later on!
Best
Silvan
Von meinem iPhone gesendet
> Am 08.05.2015 um 18:10 schrieb Varun Vasudev <vv...@hortonworks.com>:
>
> Hi Silvan,
>
> Are you trying to run MapReduce? You could look at createApplicationSubmissionContext and createApplicationResource in YarnRunner.java. Have you overridden the resolvePath function in your filesystem implementation?
>
> -Varun
>
> From: Silvan Kaiser
> Reply-To: "user@hadoop.apache.org"
> Date: Friday, May 8, 2015 at 3:52 PM
> To: "user@hadoop.apache.org"
> Subject: URI missing scheme and authority in job start with new FileSystem implementation
>
>
> Hello!
> I am running a new FileSystem implementation, testing it with a simple standalone operation from the Hadoop "Single Node Setup" page. Staging example jobs works alright but upon scheduling the container management fails because of an URI without scheme
> (and authority).
>
> NodeManager log snippet:
> ...
>
> 2015-05-08 15:18:15,331 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Got exception parsing jobSubmitDir/job.splitmetainfo and value resource { port: -1 file: "/tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo"
> } size: 771 timestamp: 1431091092000 type: FILE visibility: APPLICATION
> 2015-05-08 15:18:15,331 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Failed to parse resource-request
> java.net.URISyntaxException: Expected scheme name at index 0: :///tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo
>
> ...
>
>
> The path element of the URI is correct, it simply is missing it's scheme and authority but i cannot find out why so far.
> Please note that the URIs used during staging are correct and contain scheme & authority correctly. This means the incomplete URI is created during job creation.
>
> My guess so far: this is either because I am missing some configuration for the Nodemanager (therefore he does not know about my FileSystem implementation and throws the unknown scheme & authority out the window) or that the FileSystem implementation has
> an error / did not override a method that requires adoption.
>
> Can someone please give me a hint where the URIs used by the NodeManager are constructed prior to scheduling? This would allow me to trace back where the incomplete URI is coming from...
>
> Best regards
> Silvan
>
>
> --
> Quobyte GmbH
> Boyenstr. 41 - 10115 Berlin-Mitte - Germany
> +49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/><http://www.quobyte.com/>
> Amtsgericht Berlin-Charlottenburg, HRB 149012B
> Management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
>
>
>
>
>
>
>
>
>
> --
> Quobyte GmbH
> Hardenbergplatz 2 - 10623 Berlin - Germany
> +49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/>Amtsgericht Berlin-Charlottenburg, HRB 149012B
> management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
--
--
*Quobyte* GmbH
Hardenbergplatz 2 - 10623 Berlin - Germany
+49-30-814 591 800 - www.quobyte.com
Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
Re: URI missing scheme and authority in job start with new FileSystem implementation
Posted by Silvan Kaiser <si...@quobyte.com>.
Dear Varun,
thanks for the hints! Yep it's Mapreduce.
Can't look at the code right now but I cannot remember overriding resolvePath. Will check it out and give feedback later on!
Best
Silvan
Von meinem iPhone gesendet
> Am 08.05.2015 um 18:10 schrieb Varun Vasudev <vv...@hortonworks.com>:
>
> Hi Silvan,
>
> Are you trying to run MapReduce? You could look at createApplicationSubmissionContext and createApplicationResource in YarnRunner.java. Have you overridden the resolvePath function in your filesystem implementation?
>
> -Varun
>
> From: Silvan Kaiser
> Reply-To: "user@hadoop.apache.org"
> Date: Friday, May 8, 2015 at 3:52 PM
> To: "user@hadoop.apache.org"
> Subject: URI missing scheme and authority in job start with new FileSystem implementation
>
>
> Hello!
> I am running a new FileSystem implementation, testing it with a simple standalone operation from the Hadoop "Single Node Setup" page. Staging example jobs works alright but upon scheduling the container management fails because of an URI without scheme
> (and authority).
>
> NodeManager log snippet:
> ...
>
> 2015-05-08 15:18:15,331 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Got exception parsing jobSubmitDir/job.splitmetainfo and value resource { port: -1 file: "/tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo"
> } size: 771 timestamp: 1431091092000 type: FILE visibility: APPLICATION
> 2015-05-08 15:18:15,331 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Failed to parse resource-request
> java.net.URISyntaxException: Expected scheme name at index 0: :///tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo
>
> ...
>
>
> The path element of the URI is correct, it simply is missing it's scheme and authority but i cannot find out why so far.
> Please note that the URIs used during staging are correct and contain scheme & authority correctly. This means the incomplete URI is created during job creation.
>
> My guess so far: this is either because I am missing some configuration for the Nodemanager (therefore he does not know about my FileSystem implementation and throws the unknown scheme & authority out the window) or that the FileSystem implementation has
> an error / did not override a method that requires adoption.
>
> Can someone please give me a hint where the URIs used by the NodeManager are constructed prior to scheduling? This would allow me to trace back where the incomplete URI is coming from...
>
> Best regards
> Silvan
>
>
> --
> Quobyte GmbH
> Boyenstr. 41 - 10115 Berlin-Mitte - Germany
> +49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/><http://www.quobyte.com/>
> Amtsgericht Berlin-Charlottenburg, HRB 149012B
> Management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
>
>
>
>
>
>
>
>
>
> --
> Quobyte GmbH
> Hardenbergplatz 2 - 10623 Berlin - Germany
> +49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/>Amtsgericht Berlin-Charlottenburg, HRB 149012B
> management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
--
--
*Quobyte* GmbH
Hardenbergplatz 2 - 10623 Berlin - Germany
+49-30-814 591 800 - www.quobyte.com
Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
Re: URI missing scheme and authority in job start with new FileSystem implementation
Posted by Silvan Kaiser <si...@quobyte.com>.
Dear Varun,
thanks for the hints! Yep it's Mapreduce.
Can't look at the code right now but I cannot remember overriding resolvePath. Will check it out and give feedback later on!
Best
Silvan
Von meinem iPhone gesendet
> Am 08.05.2015 um 18:10 schrieb Varun Vasudev <vv...@hortonworks.com>:
>
> Hi Silvan,
>
> Are you trying to run MapReduce? You could look at createApplicationSubmissionContext and createApplicationResource in YarnRunner.java. Have you overridden the resolvePath function in your filesystem implementation?
>
> -Varun
>
> From: Silvan Kaiser
> Reply-To: "user@hadoop.apache.org"
> Date: Friday, May 8, 2015 at 3:52 PM
> To: "user@hadoop.apache.org"
> Subject: URI missing scheme and authority in job start with new FileSystem implementation
>
>
> Hello!
> I am running a new FileSystem implementation, testing it with a simple standalone operation from the Hadoop "Single Node Setup" page. Staging example jobs works alright but upon scheduling the container management fails because of an URI without scheme
> (and authority).
>
> NodeManager log snippet:
> ...
>
> 2015-05-08 15:18:15,331 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Got exception parsing jobSubmitDir/job.splitmetainfo and value resource { port: -1 file: "/tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo"
> } size: 771 timestamp: 1431091092000 type: FILE visibility: APPLICATION
> 2015-05-08 15:18:15,331 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Failed to parse resource-request
> java.net.URISyntaxException: Expected scheme name at index 0: :///tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo
>
> ...
>
>
> The path element of the URI is correct, it simply is missing it's scheme and authority but i cannot find out why so far.
> Please note that the URIs used during staging are correct and contain scheme & authority correctly. This means the incomplete URI is created during job creation.
>
> My guess so far: this is either because I am missing some configuration for the Nodemanager (therefore he does not know about my FileSystem implementation and throws the unknown scheme & authority out the window) or that the FileSystem implementation has
> an error / did not override a method that requires adoption.
>
> Can someone please give me a hint where the URIs used by the NodeManager are constructed prior to scheduling? This would allow me to trace back where the incomplete URI is coming from...
>
> Best regards
> Silvan
>
>
> --
> Quobyte GmbH
> Boyenstr. 41 - 10115 Berlin-Mitte - Germany
> +49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/><http://www.quobyte.com/>
> Amtsgericht Berlin-Charlottenburg, HRB 149012B
> Management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
>
>
>
>
>
>
>
>
>
> --
> Quobyte GmbH
> Hardenbergplatz 2 - 10623 Berlin - Germany
> +49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/>Amtsgericht Berlin-Charlottenburg, HRB 149012B
> management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
--
--
*Quobyte* GmbH
Hardenbergplatz 2 - 10623 Berlin - Germany
+49-30-814 591 800 - www.quobyte.com
Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
Re: URI missing scheme and authority in job start with new FileSystem implementation
Posted by Silvan Kaiser <si...@quobyte.com>.
Dear Varun,
thanks for the hints! Yep it's Mapreduce.
Can't look at the code right now but I cannot remember overriding resolvePath. Will check it out and give feedback later on!
Best
Silvan
Von meinem iPhone gesendet
> Am 08.05.2015 um 18:10 schrieb Varun Vasudev <vv...@hortonworks.com>:
>
> Hi Silvan,
>
> Are you trying to run MapReduce? You could look at createApplicationSubmissionContext and createApplicationResource in YarnRunner.java. Have you overridden the resolvePath function in your filesystem implementation?
>
> -Varun
>
> From: Silvan Kaiser
> Reply-To: "user@hadoop.apache.org"
> Date: Friday, May 8, 2015 at 3:52 PM
> To: "user@hadoop.apache.org"
> Subject: URI missing scheme and authority in job start with new FileSystem implementation
>
>
> Hello!
> I am running a new FileSystem implementation, testing it with a simple standalone operation from the Hadoop "Single Node Setup" page. Staging example jobs works alright but upon scheduling the container management fails because of an URI without scheme
> (and authority).
>
> NodeManager log snippet:
> ...
>
> 2015-05-08 15:18:15,331 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Got exception parsing jobSubmitDir/job.splitmetainfo and value resource { port: -1 file: "/tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo"
> } size: 771 timestamp: 1431091092000 type: FILE visibility: APPLICATION
> 2015-05-08 15:18:15,331 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Failed to parse resource-request
> java.net.URISyntaxException: Expected scheme name at index 0: :///tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo
>
> ...
>
>
> The path element of the URI is correct, it simply is missing it's scheme and authority but i cannot find out why so far.
> Please note that the URIs used during staging are correct and contain scheme & authority correctly. This means the incomplete URI is created during job creation.
>
> My guess so far: this is either because I am missing some configuration for the Nodemanager (therefore he does not know about my FileSystem implementation and throws the unknown scheme & authority out the window) or that the FileSystem implementation has
> an error / did not override a method that requires adoption.
>
> Can someone please give me a hint where the URIs used by the NodeManager are constructed prior to scheduling? This would allow me to trace back where the incomplete URI is coming from...
>
> Best regards
> Silvan
>
>
> --
> Quobyte GmbH
> Boyenstr. 41 - 10115 Berlin-Mitte - Germany
> +49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/><http://www.quobyte.com/>
> Amtsgericht Berlin-Charlottenburg, HRB 149012B
> Management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
>
>
>
>
>
>
>
>
>
> --
> Quobyte GmbH
> Hardenbergplatz 2 - 10623 Berlin - Germany
> +49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/>Amtsgericht Berlin-Charlottenburg, HRB 149012B
> management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
--
--
*Quobyte* GmbH
Hardenbergplatz 2 - 10623 Berlin - Germany
+49-30-814 591 800 - www.quobyte.com
Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
Re: URI missing scheme and authority in job start with new
FileSystem implementation
Posted by Varun Vasudev <vv...@hortonworks.com>.
Hi Silvan,
Are you trying to run MapReduce? You could look at createApplicationSubmissionContext and createApplicationResource in YarnRunner.java. Have you overridden the resolvePath function in your filesystem implementation?
-Varun
From: Silvan Kaiser
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>"
Date: Friday, May 8, 2015 at 3:52 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>"
Subject: URI missing scheme and authority in job start with new FileSystem implementation
Hello!
I am running a new FileSystem implementation, testing it with a simple standalone operation from the Hadoop "Single Node Setup" page. Staging example jobs works alright but upon scheduling the container management fails because of an URI without scheme
(and authority).
NodeManager log snippet:
...
2015-05-08 15:18:15,331 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Got exception parsing jobSubmitDir/job.splitmetainfo and value resource { port: -1 file: "/tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo"
} size: 771 timestamp: 1431091092000 type: FILE visibility: APPLICATION
2015-05-08 15:18:15,331 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Failed to parse resource-request
java.net.URISyntaxException: Expected scheme name at index 0: :///tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo
...
The path element of the URI is correct, it simply is missing it's scheme and authority but i cannot find out why so far.
Please note that the URIs used during staging are correct and contain scheme & authority correctly. This means the incomplete URI is created during job creation.
My guess so far: this is either because I am missing some configuration for the Nodemanager (therefore he does not know about my FileSystem implementation and throws the unknown scheme & authority out the window) or that the FileSystem implementation has
an error / did not override a method that requires adoption.
Can someone please give me a hint where the URIs used by the NodeManager are constructed prior to scheduling? This would allow me to trace back where the incomplete URI is coming from...
Best regards
Silvan
--
Quobyte GmbH
Boyenstr. 41 - 10115 Berlin-Mitte - Germany
+49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/><http://www.quobyte.com/>
Amtsgericht Berlin-Charlottenburg, HRB 149012B
Management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
--
Quobyte GmbH
Hardenbergplatz 2 - 10623 Berlin - Germany
+49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/>Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
Re: URI missing scheme and authority in job start with new
FileSystem implementation
Posted by Varun Vasudev <vv...@hortonworks.com>.
Hi Silvan,
Are you trying to run MapReduce? You could look at createApplicationSubmissionContext and createApplicationResource in YarnRunner.java. Have you overridden the resolvePath function in your filesystem implementation?
-Varun
From: Silvan Kaiser
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>"
Date: Friday, May 8, 2015 at 3:52 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>"
Subject: URI missing scheme and authority in job start with new FileSystem implementation
Hello!
I am running a new FileSystem implementation, testing it with a simple standalone operation from the Hadoop "Single Node Setup" page. Staging example jobs works alright but upon scheduling the container management fails because of an URI without scheme
(and authority).
NodeManager log snippet:
...
2015-05-08 15:18:15,331 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Got exception parsing jobSubmitDir/job.splitmetainfo and value resource { port: -1 file: "/tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo"
} size: 771 timestamp: 1431091092000 type: FILE visibility: APPLICATION
2015-05-08 15:18:15,331 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Failed to parse resource-request
java.net.URISyntaxException: Expected scheme name at index 0: :///tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo
...
The path element of the URI is correct, it simply is missing it's scheme and authority but i cannot find out why so far.
Please note that the URIs used during staging are correct and contain scheme & authority correctly. This means the incomplete URI is created during job creation.
My guess so far: this is either because I am missing some configuration for the Nodemanager (therefore he does not know about my FileSystem implementation and throws the unknown scheme & authority out the window) or that the FileSystem implementation has
an error / did not override a method that requires adoption.
Can someone please give me a hint where the URIs used by the NodeManager are constructed prior to scheduling? This would allow me to trace back where the incomplete URI is coming from...
Best regards
Silvan
--
Quobyte GmbH
Boyenstr. 41 - 10115 Berlin-Mitte - Germany
+49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/><http://www.quobyte.com/>
Amtsgericht Berlin-Charlottenburg, HRB 149012B
Management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
--
Quobyte GmbH
Hardenbergplatz 2 - 10623 Berlin - Germany
+49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/>Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
Re: URI missing scheme and authority in job start with new
FileSystem implementation
Posted by Varun Vasudev <vv...@hortonworks.com>.
Hi Silvan,
Are you trying to run MapReduce? You could look at createApplicationSubmissionContext and createApplicationResource in YarnRunner.java. Have you overridden the resolvePath function in your filesystem implementation?
-Varun
From: Silvan Kaiser
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>"
Date: Friday, May 8, 2015 at 3:52 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>"
Subject: URI missing scheme and authority in job start with new FileSystem implementation
Hello!
I am running a new FileSystem implementation, testing it with a simple standalone operation from the Hadoop "Single Node Setup" page. Staging example jobs works alright but upon scheduling the container management fails because of an URI without scheme
(and authority).
NodeManager log snippet:
...
2015-05-08 15:18:15,331 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Got exception parsing jobSubmitDir/job.splitmetainfo and value resource { port: -1 file: "/tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo"
} size: 771 timestamp: 1431091092000 type: FILE visibility: APPLICATION
2015-05-08 15:18:15,331 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Failed to parse resource-request
java.net.URISyntaxException: Expected scheme name at index 0: :///tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo
...
The path element of the URI is correct, it simply is missing it's scheme and authority but i cannot find out why so far.
Please note that the URIs used during staging are correct and contain scheme & authority correctly. This means the incomplete URI is created during job creation.
My guess so far: this is either because I am missing some configuration for the Nodemanager (therefore he does not know about my FileSystem implementation and throws the unknown scheme & authority out the window) or that the FileSystem implementation has
an error / did not override a method that requires adoption.
Can someone please give me a hint where the URIs used by the NodeManager are constructed prior to scheduling? This would allow me to trace back where the incomplete URI is coming from...
Best regards
Silvan
--
Quobyte GmbH
Boyenstr. 41 - 10115 Berlin-Mitte - Germany
+49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/><http://www.quobyte.com/>
Amtsgericht Berlin-Charlottenburg, HRB 149012B
Management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
--
Quobyte GmbH
Hardenbergplatz 2 - 10623 Berlin - Germany
+49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/>Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
Re: URI missing scheme and authority in job start with new
FileSystem implementation
Posted by Varun Vasudev <vv...@hortonworks.com>.
Hi Silvan,
Are you trying to run MapReduce? You could look at createApplicationSubmissionContext and createApplicationResource in YarnRunner.java. Have you overridden the resolvePath function in your filesystem implementation?
-Varun
From: Silvan Kaiser
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>"
Date: Friday, May 8, 2015 at 3:52 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>"
Subject: URI missing scheme and authority in job start with new FileSystem implementation
Hello!
I am running a new FileSystem implementation, testing it with a simple standalone operation from the Hadoop "Single Node Setup" page. Staging example jobs works alright but upon scheduling the container management fails because of an URI without scheme
(and authority).
NodeManager log snippet:
...
2015-05-08 15:18:15,331 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Got exception parsing jobSubmitDir/job.splitmetainfo and value resource { port: -1 file: "/tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo"
} size: 771 timestamp: 1431091092000 type: FILE visibility: APPLICATION
2015-05-08 15:18:15,331 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Failed to parse resource-request
java.net.URISyntaxException: Expected scheme name at index 0: :///tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo
...
The path element of the URI is correct, it simply is missing it's scheme and authority but i cannot find out why so far.
Please note that the URIs used during staging are correct and contain scheme & authority correctly. This means the incomplete URI is created during job creation.
My guess so far: this is either because I am missing some configuration for the Nodemanager (therefore he does not know about my FileSystem implementation and throws the unknown scheme & authority out the window) or that the FileSystem implementation has
an error / did not override a method that requires adoption.
Can someone please give me a hint where the URIs used by the NodeManager are constructed prior to scheduling? This would allow me to trace back where the incomplete URI is coming from...
Best regards
Silvan
--
Quobyte GmbH
Boyenstr. 41 - 10115 Berlin-Mitte - Germany
+49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/><http://www.quobyte.com/>
Amtsgericht Berlin-Charlottenburg, HRB 149012B
Management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
--
Quobyte GmbH
Hardenbergplatz 2 - 10623 Berlin - Germany
+49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/>Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender