You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Lukas Kairies <lu...@googlemail.com> on 2014/02/03 14:33:23 UTC
Re: Invalide URI in job start
Hello,
first thanks for your reply and sorry for my late answer. The JobClient
prints the following URL after I added a system.out statement:
port: -1 file:
"/tmp/hadoop-yarn/staging/lkairies/.staging/job_1391433848658_0001/job.jar"
(for all job files, i.e. also for the job.xml,...).
So there is no scheme and the port is also not set. The same URL is
logged in the node manager log file.
Thanks,
Lukas Kairies
Am 27.01.2014 20:00, schrieb Vinod Kumar Vavilapalli:
> Need your help to debug this. Seems like the scheme is getting lost
> somewhere along the way. Clearly as you say if job.jar is on the
> file-system, then JobClient is properly uploading it. There are
> multilple things that you'll need to check
> - Check the NodeManager logs for the URL. It does print what URL it
> is trying to download from. Check if the scheme is getting there or not.
> - If that doesn't tell you something, change JobClient to print the
> URL before it constructs the ContainerLaunchContext for the
> ApplicationMaster. You'll need to do this in YarnRunner.java.
> Specifically the method createApplicationResource.
>
> Thanks,
> +Vinod Kumar Vavilapalli
> Hortonworks Inc.
> http://hortonworks.com/
>
> On Jan 27, 2014, at 2:05 AM, Lukas Kairies
> <lukas.xtreemfs@googlemail.com <ma...@googlemail.com>>
> wrote:
>
>> Hello,
>>
>> I try to use XtreemFS as an alternative file system for Hadoop 2.x.
>> There is an existing FileSystem implementation for Hadoop 1.x that
>> works fine. First think I did was to implement a DelegateToFileSystem
>> subclass to provide an AbstractFileSystem implementation for XtreemFS
>> (just constructors that use the FileSystem implementation). When I
>> start the wordcount example application I get the following Exception
>> on the NodeManager:
>>
>> 2014-01-20 14:18:19,349 WARN
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>> Failed to parse resource-request
>> java.net.URISyntaxException: Expected scheme name at index 0:
>> :///tmp/hadoop-yarn/staging/lkairies/.staging/job_1390223418764_0004/job.jar
>> at java.net.URI$Parser.fail(URI.java:2829)
>> at java.net.URI$Parser.failExpecting(URI.java:2835)
>> at java.net.URI$Parser.parse(URI.java:3027)
>> at java.net.URI.<init>(URI.java:753)
>> at
>> org.apache.hadoop.yarn.util.ConverterUtils.getPathFromYarnURL(ConverterUtils.java:80)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourceRequest.<init>(LocalResourceRequest.java:46)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$RequestResourcesTransition.transition(ContainerImpl.java:529)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$RequestResourcesTransition.transition(ContainerImpl.java:497)
>> at
>> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
>> at
>> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
>> at
>> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
>> at
>> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:864)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:73)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:815)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:808)
>> at
>> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
>> at
>> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:81)
>> at java.lang.Thread.run(Thread.java:724)
>>
>> Additionally the following is printed on the console:
>>
>> 14/01/27 11:02:14 INFO input.FileInputFormat: Total input paths to
>> process : 1
>> 14/01/27 11:02:14 INFO mapreduce.JobSubmitter: number of splits:1
>> 14/01/27 11:02:15 INFO Configuration.deprecation: user.name is
>> deprecated. Instead, use mapreduce.job.user.name
>> 14/01/27 11:02:15 INFO Configuration.deprecation: mapred.jar is
>> deprecated. Instead, use mapreduce.job.jar
>> 14/01/27 11:02:15 INFO Configuration.deprecation:
>> mapred.output.value.class is deprecated. Instead, use
>> mapreduce.job.output.value.class
>> 14/01/27 11:02:15 INFO Configuration.deprecation:
>> mapreduce.combine.class is deprecated. Instead, use
>> mapreduce.job.combine.class
>> 14/01/27 11:02:15 INFO Configuration.deprecation: mapreduce.map.class
>> is deprecated. Instead, use mapreduce.job.map.class
>> 14/01/27 11:02:15 INFO Configuration.deprecation: mapred.job.name is
>> deprecated. Instead, use mapreduce.job.name
>> 14/01/27 11:02:15 INFO Configuration.deprecation:
>> mapreduce.reduce.class is deprecated. Instead, use
>> mapreduce.job.reduce.class
>> 14/01/27 11:02:15 INFO Configuration.deprecation: mapred.input.dir is
>> deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
>> 14/01/27 11:02:15 INFO Configuration.deprecation: mapred.output.dir
>> is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
>> 14/01/27 11:02:15 INFO Configuration.deprecation: mapred.map.tasks is
>> deprecated. Instead, use mapreduce.job.maps
>> 14/01/27 11:02:15 INFO Configuration.deprecation:
>> mapred.output.key.class is deprecated. Instead, use
>> mapreduce.job.output.key.class
>> 14/01/27 11:02:15 INFO Configuration.deprecation: mapred.working.dir
>> is deprecated. Instead, use mapreduce.job.working.dir
>> 14/01/27 11:02:15 INFO mapreduce.JobSubmitter: Submitting tokens for
>> job: job_1390816735288_0001
>> 14/01/27 11:02:15 INFO impl.YarnClientImpl: Submitted application
>> application_1390816735288_0001 to ResourceManager at /0.0.0.0:8032
>> 14/01/27 11:02:15 INFO mapreduce.Job: The url to track the job:
>> http://ludiwg:8088/proxy/application_1390816735288_0001/
>> 14/01/27 11:02:15 INFO mapreduce.Job: Running job: job_1390816735288_0001
>> 14/01/27 11:02:19 INFO mapreduce.Job: Job job_1390816735288_0001
>> running in uber mode : false
>> 14/01/27 11:02:19 INFO mapreduce.Job: map 0% reduce 0%
>> 14/01/27 11:02:19 INFO mapreduce.Job: Job job_1390816735288_0001
>> failed with state FAILED due to: Application
>> application_1390816735288_0001 failed 2 times due to AM Container for
>> appattempt_1390816735288_0001_000002 exited with exitCode: -1000 due
>> to: .Failing this attempt.. Failing the application.
>> 14/01/27 11:02:19 INFO mapreduce.Job: Counters: 0
>>
>>
>> The job files are created in XtreemFS. After a lot of debugging I
>> still did not find the problem.
>>
>> Any idea how to fix this?
>>
>> Regards,
>> Lukas
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or
> entity to which it is addressed and may contain information that is
> confidential, privileged and exempt from disclosure under applicable
> law. If the reader of this message is not the intended recipient, you
> are hereby notified that any printing, copying, dissemination,
> distribution, disclosure or forwarding of this communication is
> strictly prohibited. If you have received this communication in error,
> please contact the sender immediately and delete it from your system.
> Thank You.