You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Matthias Kricke <ma...@gmail.com> on 2012/08/22 18:09:26 UTC
set replication of output to 1 in CDH3
Dear mailing list,
I want to set the replication of my output file to 1 for clouderas hadoop
3. Some web research told me to do the following:
hadoop jar fu.jar com.bar -D dfs.replication=1 input output
I adapted it a bit, because my jar has a specified main class. I do
something like that:
hadoop jar fu.jar -D dfs.replication=1 input output
When I now execute this, I get the following error:
12/08/22 18:04:42 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
12/08/22 18:04:43 INFO mapred.JobClient: Cleaning up the staging area
someServerAndFileAdress
Exception in thread "main"
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path
does not exist: someServerAndFileAdress*/-D*
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:231)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:248)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:944)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961)
at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
at
com.unister.semweb.preprocessingMapperStyle.core.StartMapper.main(StartMapper.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
I would appreciate your help.
Greetings,
MK
Re: set replication of output to 1 in CDH3
Posted by Harsh J <ha...@cloudera.com>.
Hi,
This is your issue:
> 12/08/22 18:04:42 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
For using the -D parameters automatically, your application needs to
implement the Tool interface. See
http://archive.cloudera.com/cdh/3/hadoop/api/org/apache/hadoop/util/Tool.html
for the API and an example.
On Wed, Aug 22, 2012 at 9:39 PM, Matthias Kricke
<ma...@gmail.com> wrote:
> Dear mailing list,
>
> I want to set the replication of my output file to 1 for clouderas hadoop 3.
> Some web research told me to do the following:
> hadoop jar fu.jar com.bar -D dfs.replication=1 input output
>
> I adapted it a bit, because my jar has a specified main class. I do
> something like that:
> hadoop jar fu.jar -D dfs.replication=1 input output
>
> When I now execute this, I get the following error:
>
> 12/08/22 18:04:42 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 12/08/22 18:04:43 INFO mapred.JobClient: Cleaning up the staging area
> someServerAndFileAdress
> Exception in thread "main"
> org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does
> not exist: someServerAndFileAdress/-D
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:231)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:248)
> at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:944)
> at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961)
> at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
> at
> com.unister.semweb.preprocessingMapperStyle.core.StartMapper.main(StartMapper.java:43)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>
> I would appreciate your help.
>
> Greetings,
> MK
>
--
Harsh J
Re: set replication of output to 1 in CDH3
Posted by Harsh J <ha...@cloudera.com>.
Hi,
This is your issue:
> 12/08/22 18:04:42 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
For using the -D parameters automatically, your application needs to
implement the Tool interface. See
http://archive.cloudera.com/cdh/3/hadoop/api/org/apache/hadoop/util/Tool.html
for the API and an example.
On Wed, Aug 22, 2012 at 9:39 PM, Matthias Kricke
<ma...@gmail.com> wrote:
> Dear mailing list,
>
> I want to set the replication of my output file to 1 for clouderas hadoop 3.
> Some web research told me to do the following:
> hadoop jar fu.jar com.bar -D dfs.replication=1 input output
>
> I adapted it a bit, because my jar has a specified main class. I do
> something like that:
> hadoop jar fu.jar -D dfs.replication=1 input output
>
> When I now execute this, I get the following error:
>
> 12/08/22 18:04:42 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 12/08/22 18:04:43 INFO mapred.JobClient: Cleaning up the staging area
> someServerAndFileAdress
> Exception in thread "main"
> org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does
> not exist: someServerAndFileAdress/-D
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:231)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:248)
> at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:944)
> at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961)
> at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
> at
> com.unister.semweb.preprocessingMapperStyle.core.StartMapper.main(StartMapper.java:43)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>
> I would appreciate your help.
>
> Greetings,
> MK
>
--
Harsh J
Re: set replication of output to 1 in CDH3
Posted by Harsh J <ha...@cloudera.com>.
Hi,
This is your issue:
> 12/08/22 18:04:42 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
For using the -D parameters automatically, your application needs to
implement the Tool interface. See
http://archive.cloudera.com/cdh/3/hadoop/api/org/apache/hadoop/util/Tool.html
for the API and an example.
On Wed, Aug 22, 2012 at 9:39 PM, Matthias Kricke
<ma...@gmail.com> wrote:
> Dear mailing list,
>
> I want to set the replication of my output file to 1 for clouderas hadoop 3.
> Some web research told me to do the following:
> hadoop jar fu.jar com.bar -D dfs.replication=1 input output
>
> I adapted it a bit, because my jar has a specified main class. I do
> something like that:
> hadoop jar fu.jar -D dfs.replication=1 input output
>
> When I now execute this, I get the following error:
>
> 12/08/22 18:04:42 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 12/08/22 18:04:43 INFO mapred.JobClient: Cleaning up the staging area
> someServerAndFileAdress
> Exception in thread "main"
> org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does
> not exist: someServerAndFileAdress/-D
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:231)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:248)
> at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:944)
> at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961)
> at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
> at
> com.unister.semweb.preprocessingMapperStyle.core.StartMapper.main(StartMapper.java:43)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>
> I would appreciate your help.
>
> Greetings,
> MK
>
--
Harsh J
Re: set replication of output to 1 in CDH3
Posted by Harsh J <ha...@cloudera.com>.
Hi,
This is your issue:
> 12/08/22 18:04:42 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
For using the -D parameters automatically, your application needs to
implement the Tool interface. See
http://archive.cloudera.com/cdh/3/hadoop/api/org/apache/hadoop/util/Tool.html
for the API and an example.
On Wed, Aug 22, 2012 at 9:39 PM, Matthias Kricke
<ma...@gmail.com> wrote:
> Dear mailing list,
>
> I want to set the replication of my output file to 1 for clouderas hadoop 3.
> Some web research told me to do the following:
> hadoop jar fu.jar com.bar -D dfs.replication=1 input output
>
> I adapted it a bit, because my jar has a specified main class. I do
> something like that:
> hadoop jar fu.jar -D dfs.replication=1 input output
>
> When I now execute this, I get the following error:
>
> 12/08/22 18:04:42 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 12/08/22 18:04:43 INFO mapred.JobClient: Cleaning up the staging area
> someServerAndFileAdress
> Exception in thread "main"
> org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does
> not exist: someServerAndFileAdress/-D
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:231)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:248)
> at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:944)
> at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961)
> at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
> at
> com.unister.semweb.preprocessingMapperStyle.core.StartMapper.main(StartMapper.java:43)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>
> I would appreciate your help.
>
> Greetings,
> MK
>
--
Harsh J