You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by xeonmailinglist <xe...@gmail.com> on 2015/02/24 19:57:21 UTC
"No FileSystem for scheme: hdfs" when submitting a remote job
Hi,
I am trying to submit a remote mapreduce job in Yarn with the
configuration below, but I get the error |No FileSystem for scheme:
hdfs|. The error is so also below. Why this is happening?
| String host = "192.168.56.100";
Configuration conf = job.getConfiguration();
// this should be like defined in your yarn-site.xml
conf.set("yarn.resourcemanager.address", host + ":50001");
// framework is now "yarn", should be defined like this in mapred-site.xm
conf.set("mapreduce.framework.name", "yarn");
// like defined in hdfs-site.xml
conf.set("fs.defaultFS", "hdfs://" + host + ":9000");
for (Path inputPath : inputPaths)
FileInputFormat.addInputPath(job, new Path(conf.get("fs.defaultFS") + "/" + inputPath.toString()));
FileOutputFormat.setOutputPath(job, new Path(conf.get("fs.defaultFS") + "/" + tempPath));
job.waitForCompletion(true);
|
|Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518)
at org.apache.hadoop.mapred.examples.JobExecution.submit(JobExecution.java:169)
at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101)
|
Thanks,
Re: "No FileSystem for scheme: hdfs" when submitting a remote job
Posted by Xuan Gong <xg...@hortonworks.com>.
The jar file will be uploaded into HDFS which could be shared.
From: xeonmailinglist <xe...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 24, 2015 at 11:23 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: "No FileSystem for scheme: hdfs" when submitting a remote job
Yes, this was the problem. But I have another question:
When I am starting a remote job, the job will run in the remote resource manager. But how the job that is running remotely will know where are the map and reduce functions?
On 24-02-2015 19:03, Xuan Gong wrote:
Have you added hadoop-hdfs-xxxx.jar into your class path ?
From: xeonmailinglist <xe...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 24, 2015 at 10:57 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: "No FileSystem for scheme: hdfs" when submitting a remote job
Hi,
I am trying to submit a remote mapreduce job in Yarn with the configuration below, but I get the error No FileSystem for scheme: hdfs. The error is so also below. Why this is happening?
String host = "192.168.56.100";
Configuration conf = job.getConfiguration();
// this should be like defined in your yarn-site.xml
conf.set("yarn.resourcemanager.address", host + ":50001");
// framework is now "yarn", should be defined like this in mapred-site.xm
conf.set("mapreduce.framework.name", "yarn");
// like defined in hdfs-site.xml
conf.set("fs.defaultFS", "hdfs://" + host + ":9000");
for (Path inputPath : inputPaths)
FileInputFormat.addInputPath(job, new Path(conf.get("fs.defaultFS") + "/" + inputPath.toString()));
FileOutputFormat.setOutputPath(job, new Path(conf.get("fs.defaultFS") + "/" + tempPath));
job.waitForCompletion(true);
Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518)
at org.apache.hadoop.mapred.examples.JobExecution.submit(JobExecution.java:169)
at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101)
Thanks,
Re: "No FileSystem for scheme: hdfs" when submitting a remote job
Posted by Xuan Gong <xg...@hortonworks.com>.
The jar file will be uploaded into HDFS which could be shared.
From: xeonmailinglist <xe...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 24, 2015 at 11:23 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: "No FileSystem for scheme: hdfs" when submitting a remote job
Yes, this was the problem. But I have another question:
When I am starting a remote job, the job will run in the remote resource manager. But how the job that is running remotely will know where are the map and reduce functions?
On 24-02-2015 19:03, Xuan Gong wrote:
Have you added hadoop-hdfs-xxxx.jar into your class path ?
From: xeonmailinglist <xe...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 24, 2015 at 10:57 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: "No FileSystem for scheme: hdfs" when submitting a remote job
Hi,
I am trying to submit a remote mapreduce job in Yarn with the configuration below, but I get the error No FileSystem for scheme: hdfs. The error is so also below. Why this is happening?
String host = "192.168.56.100";
Configuration conf = job.getConfiguration();
// this should be like defined in your yarn-site.xml
conf.set("yarn.resourcemanager.address", host + ":50001");
// framework is now "yarn", should be defined like this in mapred-site.xm
conf.set("mapreduce.framework.name", "yarn");
// like defined in hdfs-site.xml
conf.set("fs.defaultFS", "hdfs://" + host + ":9000");
for (Path inputPath : inputPaths)
FileInputFormat.addInputPath(job, new Path(conf.get("fs.defaultFS") + "/" + inputPath.toString()));
FileOutputFormat.setOutputPath(job, new Path(conf.get("fs.defaultFS") + "/" + tempPath));
job.waitForCompletion(true);
Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518)
at org.apache.hadoop.mapred.examples.JobExecution.submit(JobExecution.java:169)
at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101)
Thanks,
Re: "No FileSystem for scheme: hdfs" when submitting a remote job
Posted by Xuan Gong <xg...@hortonworks.com>.
The jar file will be uploaded into HDFS which could be shared.
From: xeonmailinglist <xe...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 24, 2015 at 11:23 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: "No FileSystem for scheme: hdfs" when submitting a remote job
Yes, this was the problem. But I have another question:
When I am starting a remote job, the job will run in the remote resource manager. But how the job that is running remotely will know where are the map and reduce functions?
On 24-02-2015 19:03, Xuan Gong wrote:
Have you added hadoop-hdfs-xxxx.jar into your class path ?
From: xeonmailinglist <xe...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 24, 2015 at 10:57 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: "No FileSystem for scheme: hdfs" when submitting a remote job
Hi,
I am trying to submit a remote mapreduce job in Yarn with the configuration below, but I get the error No FileSystem for scheme: hdfs. The error is so also below. Why this is happening?
String host = "192.168.56.100";
Configuration conf = job.getConfiguration();
// this should be like defined in your yarn-site.xml
conf.set("yarn.resourcemanager.address", host + ":50001");
// framework is now "yarn", should be defined like this in mapred-site.xm
conf.set("mapreduce.framework.name", "yarn");
// like defined in hdfs-site.xml
conf.set("fs.defaultFS", "hdfs://" + host + ":9000");
for (Path inputPath : inputPaths)
FileInputFormat.addInputPath(job, new Path(conf.get("fs.defaultFS") + "/" + inputPath.toString()));
FileOutputFormat.setOutputPath(job, new Path(conf.get("fs.defaultFS") + "/" + tempPath));
job.waitForCompletion(true);
Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518)
at org.apache.hadoop.mapred.examples.JobExecution.submit(JobExecution.java:169)
at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101)
Thanks,
Re: "No FileSystem for scheme: hdfs" when submitting a remote job
Posted by Xuan Gong <xg...@hortonworks.com>.
The jar file will be uploaded into HDFS which could be shared.
From: xeonmailinglist <xe...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 24, 2015 at 11:23 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: "No FileSystem for scheme: hdfs" when submitting a remote job
Yes, this was the problem. But I have another question:
When I am starting a remote job, the job will run in the remote resource manager. But how the job that is running remotely will know where are the map and reduce functions?
On 24-02-2015 19:03, Xuan Gong wrote:
Have you added hadoop-hdfs-xxxx.jar into your class path ?
From: xeonmailinglist <xe...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 24, 2015 at 10:57 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: "No FileSystem for scheme: hdfs" when submitting a remote job
Hi,
I am trying to submit a remote mapreduce job in Yarn with the configuration below, but I get the error No FileSystem for scheme: hdfs. The error is so also below. Why this is happening?
String host = "192.168.56.100";
Configuration conf = job.getConfiguration();
// this should be like defined in your yarn-site.xml
conf.set("yarn.resourcemanager.address", host + ":50001");
// framework is now "yarn", should be defined like this in mapred-site.xm
conf.set("mapreduce.framework.name", "yarn");
// like defined in hdfs-site.xml
conf.set("fs.defaultFS", "hdfs://" + host + ":9000");
for (Path inputPath : inputPaths)
FileInputFormat.addInputPath(job, new Path(conf.get("fs.defaultFS") + "/" + inputPath.toString()));
FileOutputFormat.setOutputPath(job, new Path(conf.get("fs.defaultFS") + "/" + tempPath));
job.waitForCompletion(true);
Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518)
at org.apache.hadoop.mapred.examples.JobExecution.submit(JobExecution.java:169)
at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101)
Thanks,
Re: "No FileSystem for scheme: hdfs" when submitting a remote job
Posted by xeonmailinglist <xe...@gmail.com>.
Yes, this was the problem. But I have another question:
When I am starting a remote job, the job will run in the remote resource
manager. But how the job that is running remotely will know where are
the map and reduce functions?
On 24-02-2015 19:03, Xuan Gong wrote:
> Have you added hadoop-hdfs-xxxx.jar into your class path ?
>
> From: xeonmailinglist <xeonmailinglist@gmail.com
> <ma...@gmail.com>>
> Reply-To: "user@hadoop.apache.org <ma...@hadoop.apache.org>"
> <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Date: Tuesday, February 24, 2015 at 10:57 AM
> To: "user@hadoop.apache.org <ma...@hadoop.apache.org>"
> <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Subject: "No FileSystem for scheme: hdfs" when submitting a remote job
>
> Hi,
>
> I am trying to submit a remote mapreduce job in Yarn with the
> configuration below, but I get the error |No FileSystem for scheme:
> hdfs|. The error is so also below. Why this is happening?
>
> | String host = "192.168.56.100";
>
> Configuration conf = job.getConfiguration();
> // this should be like defined in your yarn-site.xml
> conf.set("yarn.resourcemanager.address", host + ":50001");
>
> // framework is now "yarn", should be defined like this in mapred-site.xm
> conf.set("mapreduce.framework.name", "yarn");
>
> // like defined in hdfs-site.xml
> conf.set("fs.defaultFS", "hdfs://" + host + ":9000");
>
> for (Path inputPath : inputPaths)
> FileInputFormat.addInputPath(job, new Path(conf.get("fs.defaultFS") + "/" + inputPath.toString()));
>
> FileOutputFormat.setOutputPath(job, new Path(conf.get("fs.defaultFS") + "/" + tempPath));
>
> job.waitForCompletion(true);
> |
> |Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
> at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518)
> at org.apache.hadoop.mapred.examples.JobExecution.submit(JobExecution.java:169)
> at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101)
> |
>
> Thanks,
>
>
Re: "No FileSystem for scheme: hdfs" when submitting a remote job
Posted by xeonmailinglist <xe...@gmail.com>.
Yes, this was the problem. But I have another question:
When I am starting a remote job, the job will run in the remote resource
manager. But how the job that is running remotely will know where are
the map and reduce functions?
On 24-02-2015 19:03, Xuan Gong wrote:
> Have you added hadoop-hdfs-xxxx.jar into your class path ?
>
> From: xeonmailinglist <xeonmailinglist@gmail.com
> <ma...@gmail.com>>
> Reply-To: "user@hadoop.apache.org <ma...@hadoop.apache.org>"
> <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Date: Tuesday, February 24, 2015 at 10:57 AM
> To: "user@hadoop.apache.org <ma...@hadoop.apache.org>"
> <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Subject: "No FileSystem for scheme: hdfs" when submitting a remote job
>
> Hi,
>
> I am trying to submit a remote mapreduce job in Yarn with the
> configuration below, but I get the error |No FileSystem for scheme:
> hdfs|. The error is so also below. Why this is happening?
>
> | String host = "192.168.56.100";
>
> Configuration conf = job.getConfiguration();
> // this should be like defined in your yarn-site.xml
> conf.set("yarn.resourcemanager.address", host + ":50001");
>
> // framework is now "yarn", should be defined like this in mapred-site.xm
> conf.set("mapreduce.framework.name", "yarn");
>
> // like defined in hdfs-site.xml
> conf.set("fs.defaultFS", "hdfs://" + host + ":9000");
>
> for (Path inputPath : inputPaths)
> FileInputFormat.addInputPath(job, new Path(conf.get("fs.defaultFS") + "/" + inputPath.toString()));
>
> FileOutputFormat.setOutputPath(job, new Path(conf.get("fs.defaultFS") + "/" + tempPath));
>
> job.waitForCompletion(true);
> |
> |Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
> at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518)
> at org.apache.hadoop.mapred.examples.JobExecution.submit(JobExecution.java:169)
> at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101)
> |
>
> Thanks,
>
>
Re: "No FileSystem for scheme: hdfs" when submitting a remote job
Posted by xeonmailinglist <xe...@gmail.com>.
Yes, this was the problem. But I have another question:
When I am starting a remote job, the job will run in the remote resource
manager. But how the job that is running remotely will know where are
the map and reduce functions?
On 24-02-2015 19:03, Xuan Gong wrote:
> Have you added hadoop-hdfs-xxxx.jar into your class path ?
>
> From: xeonmailinglist <xeonmailinglist@gmail.com
> <ma...@gmail.com>>
> Reply-To: "user@hadoop.apache.org <ma...@hadoop.apache.org>"
> <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Date: Tuesday, February 24, 2015 at 10:57 AM
> To: "user@hadoop.apache.org <ma...@hadoop.apache.org>"
> <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Subject: "No FileSystem for scheme: hdfs" when submitting a remote job
>
> Hi,
>
> I am trying to submit a remote mapreduce job in Yarn with the
> configuration below, but I get the error |No FileSystem for scheme:
> hdfs|. The error is so also below. Why this is happening?
>
> | String host = "192.168.56.100";
>
> Configuration conf = job.getConfiguration();
> // this should be like defined in your yarn-site.xml
> conf.set("yarn.resourcemanager.address", host + ":50001");
>
> // framework is now "yarn", should be defined like this in mapred-site.xm
> conf.set("mapreduce.framework.name", "yarn");
>
> // like defined in hdfs-site.xml
> conf.set("fs.defaultFS", "hdfs://" + host + ":9000");
>
> for (Path inputPath : inputPaths)
> FileInputFormat.addInputPath(job, new Path(conf.get("fs.defaultFS") + "/" + inputPath.toString()));
>
> FileOutputFormat.setOutputPath(job, new Path(conf.get("fs.defaultFS") + "/" + tempPath));
>
> job.waitForCompletion(true);
> |
> |Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
> at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518)
> at org.apache.hadoop.mapred.examples.JobExecution.submit(JobExecution.java:169)
> at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101)
> |
>
> Thanks,
>
>
Re: "No FileSystem for scheme: hdfs" when submitting a remote job
Posted by xeonmailinglist <xe...@gmail.com>.
Yes, this was the problem. But I have another question:
When I am starting a remote job, the job will run in the remote resource
manager. But how the job that is running remotely will know where are
the map and reduce functions?
On 24-02-2015 19:03, Xuan Gong wrote:
> Have you added hadoop-hdfs-xxxx.jar into your class path ?
>
> From: xeonmailinglist <xeonmailinglist@gmail.com
> <ma...@gmail.com>>
> Reply-To: "user@hadoop.apache.org <ma...@hadoop.apache.org>"
> <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Date: Tuesday, February 24, 2015 at 10:57 AM
> To: "user@hadoop.apache.org <ma...@hadoop.apache.org>"
> <user@hadoop.apache.org <ma...@hadoop.apache.org>>
> Subject: "No FileSystem for scheme: hdfs" when submitting a remote job
>
> Hi,
>
> I am trying to submit a remote mapreduce job in Yarn with the
> configuration below, but I get the error |No FileSystem for scheme:
> hdfs|. The error is so also below. Why this is happening?
>
> | String host = "192.168.56.100";
>
> Configuration conf = job.getConfiguration();
> // this should be like defined in your yarn-site.xml
> conf.set("yarn.resourcemanager.address", host + ":50001");
>
> // framework is now "yarn", should be defined like this in mapred-site.xm
> conf.set("mapreduce.framework.name", "yarn");
>
> // like defined in hdfs-site.xml
> conf.set("fs.defaultFS", "hdfs://" + host + ":9000");
>
> for (Path inputPath : inputPaths)
> FileInputFormat.addInputPath(job, new Path(conf.get("fs.defaultFS") + "/" + inputPath.toString()));
>
> FileOutputFormat.setOutputPath(job, new Path(conf.get("fs.defaultFS") + "/" + tempPath));
>
> job.waitForCompletion(true);
> |
> |Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
> at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518)
> at org.apache.hadoop.mapred.examples.JobExecution.submit(JobExecution.java:169)
> at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101)
> |
>
> Thanks,
>
>
Re: "No FileSystem for scheme: hdfs" when submitting a remote job
Posted by Xuan Gong <xg...@hortonworks.com>.
Have you added hadoop-hdfs-xxxx.jar into your class path ?
From: xeonmailinglist <xe...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 24, 2015 at 10:57 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: "No FileSystem for scheme: hdfs" when submitting a remote job
Hi,
I am trying to submit a remote mapreduce job in Yarn with the configuration below, but I get the error No FileSystem for scheme: hdfs. The error is so also below. Why this is happening?
String host = "192.168.56.100";
Configuration conf = job.getConfiguration();
// this should be like defined in your yarn-site.xml
conf.set("yarn.resourcemanager.address", host + ":50001");
// framework is now "yarn", should be defined like this in mapred-site.xm
conf.set("mapreduce.framework.name", "yarn");
// like defined in hdfs-site.xml
conf.set("fs.defaultFS", "hdfs://" + host + ":9000");
for (Path inputPath : inputPaths)
FileInputFormat.addInputPath(job, new Path(conf.get("fs.defaultFS") + "/" + inputPath.toString()));
FileOutputFormat.setOutputPath(job, new Path(conf.get("fs.defaultFS") + "/" + tempPath));
job.waitForCompletion(true);
Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518)
at org.apache.hadoop.mapred.examples.JobExecution.submit(JobExecution.java:169)
at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101)
Thanks,
Re: "No FileSystem for scheme: hdfs" when submitting a remote job
Posted by Xuan Gong <xg...@hortonworks.com>.
Have you added hadoop-hdfs-xxxx.jar into your class path ?
From: xeonmailinglist <xe...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 24, 2015 at 10:57 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: "No FileSystem for scheme: hdfs" when submitting a remote job
Hi,
I am trying to submit a remote mapreduce job in Yarn with the configuration below, but I get the error No FileSystem for scheme: hdfs. The error is so also below. Why this is happening?
String host = "192.168.56.100";
Configuration conf = job.getConfiguration();
// this should be like defined in your yarn-site.xml
conf.set("yarn.resourcemanager.address", host + ":50001");
// framework is now "yarn", should be defined like this in mapred-site.xm
conf.set("mapreduce.framework.name", "yarn");
// like defined in hdfs-site.xml
conf.set("fs.defaultFS", "hdfs://" + host + ":9000");
for (Path inputPath : inputPaths)
FileInputFormat.addInputPath(job, new Path(conf.get("fs.defaultFS") + "/" + inputPath.toString()));
FileOutputFormat.setOutputPath(job, new Path(conf.get("fs.defaultFS") + "/" + tempPath));
job.waitForCompletion(true);
Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518)
at org.apache.hadoop.mapred.examples.JobExecution.submit(JobExecution.java:169)
at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101)
Thanks,
Re: "No FileSystem for scheme: hdfs" when submitting a remote job
Posted by Xuan Gong <xg...@hortonworks.com>.
Have you added hadoop-hdfs-xxxx.jar into your class path ?
From: xeonmailinglist <xe...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 24, 2015 at 10:57 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: "No FileSystem for scheme: hdfs" when submitting a remote job
Hi,
I am trying to submit a remote mapreduce job in Yarn with the configuration below, but I get the error No FileSystem for scheme: hdfs. The error is so also below. Why this is happening?
String host = "192.168.56.100";
Configuration conf = job.getConfiguration();
// this should be like defined in your yarn-site.xml
conf.set("yarn.resourcemanager.address", host + ":50001");
// framework is now "yarn", should be defined like this in mapred-site.xm
conf.set("mapreduce.framework.name", "yarn");
// like defined in hdfs-site.xml
conf.set("fs.defaultFS", "hdfs://" + host + ":9000");
for (Path inputPath : inputPaths)
FileInputFormat.addInputPath(job, new Path(conf.get("fs.defaultFS") + "/" + inputPath.toString()));
FileOutputFormat.setOutputPath(job, new Path(conf.get("fs.defaultFS") + "/" + tempPath));
job.waitForCompletion(true);
Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518)
at org.apache.hadoop.mapred.examples.JobExecution.submit(JobExecution.java:169)
at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101)
Thanks,
Re: "No FileSystem for scheme: hdfs" when submitting a remote job
Posted by Xuan Gong <xg...@hortonworks.com>.
Have you added hadoop-hdfs-xxxx.jar into your class path ?
From: xeonmailinglist <xe...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Tuesday, February 24, 2015 at 10:57 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: "No FileSystem for scheme: hdfs" when submitting a remote job
Hi,
I am trying to submit a remote mapreduce job in Yarn with the configuration below, but I get the error No FileSystem for scheme: hdfs. The error is so also below. Why this is happening?
String host = "192.168.56.100";
Configuration conf = job.getConfiguration();
// this should be like defined in your yarn-site.xml
conf.set("yarn.resourcemanager.address", host + ":50001");
// framework is now "yarn", should be defined like this in mapred-site.xm
conf.set("mapreduce.framework.name", "yarn");
// like defined in hdfs-site.xml
conf.set("fs.defaultFS", "hdfs://" + host + ":9000");
for (Path inputPath : inputPaths)
FileInputFormat.addInputPath(job, new Path(conf.get("fs.defaultFS") + "/" + inputPath.toString()));
FileOutputFormat.setOutputPath(job, new Path(conf.get("fs.defaultFS") + "/" + tempPath));
job.waitForCompletion(true);
Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:518)
at org.apache.hadoop.mapred.examples.JobExecution.submit(JobExecution.java:169)
at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101)
Thanks,