You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Libo Yu <yu...@hotmail.com> on 2014/04/02 03:48:07 UTC
how to customize hadoop configuration for a job?
Hi all,
According to the book "Hadoop; The Definitive Guide", it is possible to use "-D property=value" to
override any default or site property in the configuration.
I gave it shot and it is true. The property specified with "-D" is ignored.
Then I put the property in an xml file and use "-conf xml_name" on the command line. But still I cannot
override the property.
The only way to override the default property is to get a Configuration reference in the code and set the property
via the reference. But that is not convenient as I need to recompile the code each time I change the property.
So what is the right way to customize the configuration for a job?
Thanks,
Libo
RE: how to customize hadoop configuration for a job?
Posted by Libo Yu <yu...@hotmail.com>.
if I remove the space, that will set system property. I pretty much followed the "
Hadoop: The definitive guide".
From: mshi@gopivotal.com
Date: Wed, 2 Apr 2014 10:14:47 +0800
Subject: Re: how to customize hadoop configuration for a job?
To: user@hadoop.apache.org
Hi Libo,
Could you post the exact command you run? Meanwhile, could you try to remove the space between -D and the "property=value", i.e. use -Dproperty=value instead of "-D property=value".
On Wed, Apr 2, 2014 at 9:48 AM, Libo Yu <yu...@hotmail.com> wrote:
Hi all,
According to the book "Hadoop; The Definitive Guide", it is possible to use "-D property=value" to
override any default or site property in the configuration.
I gave it shot and it is true. The property specified with "-D" is ignored.
Then I put the property in an xml file and use "-conf xml_name" on the command line. But still I cannot
override the property.
The only way to override the default property is to get a Configuration reference in the code and set the property
via the reference. But that is not convenient as I need to recompile the code each time I change the property.
So what is the right way to customize the configuration for a job?
Thanks,
Libo
--
Cheers
-MJ
RE: how to customize hadoop configuration for a job?
Posted by Libo Yu <yu...@hotmail.com>.
if I remove the space, that will set system property. I pretty much followed the "
Hadoop: The definitive guide".
From: mshi@gopivotal.com
Date: Wed, 2 Apr 2014 10:14:47 +0800
Subject: Re: how to customize hadoop configuration for a job?
To: user@hadoop.apache.org
Hi Libo,
Could you post the exact command you run? Meanwhile, could you try to remove the space between -D and the "property=value", i.e. use -Dproperty=value instead of "-D property=value".
On Wed, Apr 2, 2014 at 9:48 AM, Libo Yu <yu...@hotmail.com> wrote:
Hi all,
According to the book "Hadoop; The Definitive Guide", it is possible to use "-D property=value" to
override any default or site property in the configuration.
I gave it shot and it is true. The property specified with "-D" is ignored.
Then I put the property in an xml file and use "-conf xml_name" on the command line. But still I cannot
override the property.
The only way to override the default property is to get a Configuration reference in the code and set the property
via the reference. But that is not convenient as I need to recompile the code each time I change the property.
So what is the right way to customize the configuration for a job?
Thanks,
Libo
--
Cheers
-MJ
RE: how to customize hadoop configuration for a job?
Posted by Libo Yu <yu...@hotmail.com>.
That is exactly what I did. But the command line parameters are ignored.
The hadoop I am using is 2.0.0-cdh4.1.2. The property I tried is io.sort.mb.
Libo
From: unmeshabiju@gmail.com
Date: Wed, 2 Apr 2014 08:50:10 +0530
Subject: Re: how to customize hadoop configuration for a job?
To: user@hadoop.apache.org
Hi Libo, You can implement your driver code using ToolRunner.So that you can pass your extra configuration through command line instead of editing your code all the time.
Driver code
----------------public class WordCount extends Configured implements Tool {
public static void main(String[] args) throws Exception { int exitCode = ToolRunner.run(new Configuration(), new WordCount(), args);
System.exit(exitCode); } public int run(String[] args) throws Exception {
if (args.length != 2) { System.out.printf("Usage: %s [generic options] <input dir> <output dir>\n", getClass().getSimpleName());
return -1; } Job job = new Job(getConf());
job.setJarByClass(WordCount.class); job.setJobName("Word Count");
FileInputFormat.setInputPaths(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setMapperClass(WordMapper.class); job.setReducerClass(SumReducer.class);
job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class);
boolean success = job.waitForCompletion(true); return success ? 0 : 1;
}}
command line
-------------------$ hadoop jar myjar.jar MyDriver -D mapred.reduce.tasks=10 myinputdir myoutputdir
This is a better practise.
Happy Hadooping.
--
Thanks & Regards
Unmesha Sreeveni U.B
Hadoop, Bigdata DeveloperCenter for Cyber Security | Amrita Vishwa Vidyapeetham
http://www.unmeshasreeveni.blogspot.in/
RE: how to customize hadoop configuration for a job?
Posted by Libo Yu <yu...@hotmail.com>.
That is exactly what I did. But the command line parameters are ignored.
The hadoop I am using is 2.0.0-cdh4.1.2. The property I tried is io.sort.mb.
Libo
From: unmeshabiju@gmail.com
Date: Wed, 2 Apr 2014 08:50:10 +0530
Subject: Re: how to customize hadoop configuration for a job?
To: user@hadoop.apache.org
Hi Libo, You can implement your driver code using ToolRunner.So that you can pass your extra configuration through command line instead of editing your code all the time.
Driver code
----------------public class WordCount extends Configured implements Tool {
public static void main(String[] args) throws Exception { int exitCode = ToolRunner.run(new Configuration(), new WordCount(), args);
System.exit(exitCode); } public int run(String[] args) throws Exception {
if (args.length != 2) { System.out.printf("Usage: %s [generic options] <input dir> <output dir>\n", getClass().getSimpleName());
return -1; } Job job = new Job(getConf());
job.setJarByClass(WordCount.class); job.setJobName("Word Count");
FileInputFormat.setInputPaths(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setMapperClass(WordMapper.class); job.setReducerClass(SumReducer.class);
job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class);
boolean success = job.waitForCompletion(true); return success ? 0 : 1;
}}
command line
-------------------$ hadoop jar myjar.jar MyDriver -D mapred.reduce.tasks=10 myinputdir myoutputdir
This is a better practise.
Happy Hadooping.
--
Thanks & Regards
Unmesha Sreeveni U.B
Hadoop, Bigdata DeveloperCenter for Cyber Security | Amrita Vishwa Vidyapeetham
http://www.unmeshasreeveni.blogspot.in/
RE: how to customize hadoop configuration for a job?
Posted by Libo Yu <yu...@hotmail.com>.
That is exactly what I did. But the command line parameters are ignored.
The hadoop I am using is 2.0.0-cdh4.1.2. The property I tried is io.sort.mb.
Libo
From: unmeshabiju@gmail.com
Date: Wed, 2 Apr 2014 08:50:10 +0530
Subject: Re: how to customize hadoop configuration for a job?
To: user@hadoop.apache.org
Hi Libo, You can implement your driver code using ToolRunner.So that you can pass your extra configuration through command line instead of editing your code all the time.
Driver code
----------------public class WordCount extends Configured implements Tool {
public static void main(String[] args) throws Exception { int exitCode = ToolRunner.run(new Configuration(), new WordCount(), args);
System.exit(exitCode); } public int run(String[] args) throws Exception {
if (args.length != 2) { System.out.printf("Usage: %s [generic options] <input dir> <output dir>\n", getClass().getSimpleName());
return -1; } Job job = new Job(getConf());
job.setJarByClass(WordCount.class); job.setJobName("Word Count");
FileInputFormat.setInputPaths(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setMapperClass(WordMapper.class); job.setReducerClass(SumReducer.class);
job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class);
boolean success = job.waitForCompletion(true); return success ? 0 : 1;
}}
command line
-------------------$ hadoop jar myjar.jar MyDriver -D mapred.reduce.tasks=10 myinputdir myoutputdir
This is a better practise.
Happy Hadooping.
--
Thanks & Regards
Unmesha Sreeveni U.B
Hadoop, Bigdata DeveloperCenter for Cyber Security | Amrita Vishwa Vidyapeetham
http://www.unmeshasreeveni.blogspot.in/
RE: how to customize hadoop configuration for a job?
Posted by Libo Yu <yu...@hotmail.com>.
That is exactly what I did. But the command line parameters are ignored.
The hadoop I am using is 2.0.0-cdh4.1.2. The property I tried is io.sort.mb.
Libo
From: unmeshabiju@gmail.com
Date: Wed, 2 Apr 2014 08:50:10 +0530
Subject: Re: how to customize hadoop configuration for a job?
To: user@hadoop.apache.org
Hi Libo, You can implement your driver code using ToolRunner.So that you can pass your extra configuration through command line instead of editing your code all the time.
Driver code
----------------public class WordCount extends Configured implements Tool {
public static void main(String[] args) throws Exception { int exitCode = ToolRunner.run(new Configuration(), new WordCount(), args);
System.exit(exitCode); } public int run(String[] args) throws Exception {
if (args.length != 2) { System.out.printf("Usage: %s [generic options] <input dir> <output dir>\n", getClass().getSimpleName());
return -1; } Job job = new Job(getConf());
job.setJarByClass(WordCount.class); job.setJobName("Word Count");
FileInputFormat.setInputPaths(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setMapperClass(WordMapper.class); job.setReducerClass(SumReducer.class);
job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class);
boolean success = job.waitForCompletion(true); return success ? 0 : 1;
}}
command line
-------------------$ hadoop jar myjar.jar MyDriver -D mapred.reduce.tasks=10 myinputdir myoutputdir
This is a better practise.
Happy Hadooping.
--
Thanks & Regards
Unmesha Sreeveni U.B
Hadoop, Bigdata DeveloperCenter for Cyber Security | Amrita Vishwa Vidyapeetham
http://www.unmeshasreeveni.blogspot.in/
Re: how to customize hadoop configuration for a job?
Posted by unmesha sreeveni <un...@gmail.com>.
Hi Libo,
You can implement your driver code using ToolRunner.So that you can pass
your extra configuration through command line instead of editing your code
all the time.
Driver code
----------------
public class WordCount extends Configured implements Tool {
public static void main(String[] args) throws Exception {
int exitCode = ToolRunner.run(new Configuration(), new
WordCount(), args);
System.exit(exitCode);
}
public int run(String[] args) throws Exception {
if (args.length != 2) {
System.out.printf("Usage: %s [generic options] <input
dir> <output dir>\n", getClass().getSimpleName());
return -1;
}
Job job = new Job(getConf());
job.setJarByClass(WordCount.class);
job.setJobName("Word Count");
FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setMapperClass(WordMapper.class);
job.setReducerClass(SumReducer.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
boolean success = job.waitForCompletion(true);
return success ? 0 : 1;
}
}
command line
-------------------
$ hadoop jar myjar.jar MyDriver -D mapred.reduce.tasks=10 myinputdir
myoutputdir
This is a better practise.
Happy Hadooping.
--
*Thanks & Regards*
Unmesha Sreeveni U.B
Hadoop, Bigdata Developer
Center for Cyber Security | Amrita Vishwa Vidyapeetham
http://www.unmeshasreeveni.blogspot.in/
RE: how to customize hadoop configuration for a job?
Posted by Libo Yu <yu...@hotmail.com>.
if I remove the space, that will set system property. I pretty much followed the "
Hadoop: The definitive guide".
From: mshi@gopivotal.com
Date: Wed, 2 Apr 2014 10:14:47 +0800
Subject: Re: how to customize hadoop configuration for a job?
To: user@hadoop.apache.org
Hi Libo,
Could you post the exact command you run? Meanwhile, could you try to remove the space between -D and the "property=value", i.e. use -Dproperty=value instead of "-D property=value".
On Wed, Apr 2, 2014 at 9:48 AM, Libo Yu <yu...@hotmail.com> wrote:
Hi all,
According to the book "Hadoop; The Definitive Guide", it is possible to use "-D property=value" to
override any default or site property in the configuration.
I gave it shot and it is true. The property specified with "-D" is ignored.
Then I put the property in an xml file and use "-conf xml_name" on the command line. But still I cannot
override the property.
The only way to override the default property is to get a Configuration reference in the code and set the property
via the reference. But that is not convenient as I need to recompile the code each time I change the property.
So what is the right way to customize the configuration for a job?
Thanks,
Libo
--
Cheers
-MJ
RE: how to customize hadoop configuration for a job?
Posted by Libo Yu <yu...@hotmail.com>.
if I remove the space, that will set system property. I pretty much followed the "
Hadoop: The definitive guide".
From: mshi@gopivotal.com
Date: Wed, 2 Apr 2014 10:14:47 +0800
Subject: Re: how to customize hadoop configuration for a job?
To: user@hadoop.apache.org
Hi Libo,
Could you post the exact command you run? Meanwhile, could you try to remove the space between -D and the "property=value", i.e. use -Dproperty=value instead of "-D property=value".
On Wed, Apr 2, 2014 at 9:48 AM, Libo Yu <yu...@hotmail.com> wrote:
Hi all,
According to the book "Hadoop; The Definitive Guide", it is possible to use "-D property=value" to
override any default or site property in the configuration.
I gave it shot and it is true. The property specified with "-D" is ignored.
Then I put the property in an xml file and use "-conf xml_name" on the command line. But still I cannot
override the property.
The only way to override the default property is to get a Configuration reference in the code and set the property
via the reference. But that is not convenient as I need to recompile the code each time I change the property.
So what is the right way to customize the configuration for a job?
Thanks,
Libo
--
Cheers
-MJ
Re: how to customize hadoop configuration for a job?
Posted by unmesha sreeveni <un...@gmail.com>.
Hi Libo,
You can implement your driver code using ToolRunner.So that you can pass
your extra configuration through command line instead of editing your code
all the time.
Driver code
----------------
public class WordCount extends Configured implements Tool {
public static void main(String[] args) throws Exception {
int exitCode = ToolRunner.run(new Configuration(), new
WordCount(), args);
System.exit(exitCode);
}
public int run(String[] args) throws Exception {
if (args.length != 2) {
System.out.printf("Usage: %s [generic options] <input
dir> <output dir>\n", getClass().getSimpleName());
return -1;
}
Job job = new Job(getConf());
job.setJarByClass(WordCount.class);
job.setJobName("Word Count");
FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setMapperClass(WordMapper.class);
job.setReducerClass(SumReducer.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
boolean success = job.waitForCompletion(true);
return success ? 0 : 1;
}
}
command line
-------------------
$ hadoop jar myjar.jar MyDriver -D mapred.reduce.tasks=10 myinputdir
myoutputdir
This is a better practise.
Happy Hadooping.
--
*Thanks & Regards*
Unmesha Sreeveni U.B
Hadoop, Bigdata Developer
Center for Cyber Security | Amrita Vishwa Vidyapeetham
http://www.unmeshasreeveni.blogspot.in/
Re: how to customize hadoop configuration for a job?
Posted by unmesha sreeveni <un...@gmail.com>.
Hi Libo,
You can implement your driver code using ToolRunner.So that you can pass
your extra configuration through command line instead of editing your code
all the time.
Driver code
----------------
public class WordCount extends Configured implements Tool {
public static void main(String[] args) throws Exception {
int exitCode = ToolRunner.run(new Configuration(), new
WordCount(), args);
System.exit(exitCode);
}
public int run(String[] args) throws Exception {
if (args.length != 2) {
System.out.printf("Usage: %s [generic options] <input
dir> <output dir>\n", getClass().getSimpleName());
return -1;
}
Job job = new Job(getConf());
job.setJarByClass(WordCount.class);
job.setJobName("Word Count");
FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setMapperClass(WordMapper.class);
job.setReducerClass(SumReducer.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
boolean success = job.waitForCompletion(true);
return success ? 0 : 1;
}
}
command line
-------------------
$ hadoop jar myjar.jar MyDriver -D mapred.reduce.tasks=10 myinputdir
myoutputdir
This is a better practise.
Happy Hadooping.
--
*Thanks & Regards*
Unmesha Sreeveni U.B
Hadoop, Bigdata Developer
Center for Cyber Security | Amrita Vishwa Vidyapeetham
http://www.unmeshasreeveni.blogspot.in/
Re: how to customize hadoop configuration for a job?
Posted by unmesha sreeveni <un...@gmail.com>.
Hi Libo,
You can implement your driver code using ToolRunner.So that you can pass
your extra configuration through command line instead of editing your code
all the time.
Driver code
----------------
public class WordCount extends Configured implements Tool {
public static void main(String[] args) throws Exception {
int exitCode = ToolRunner.run(new Configuration(), new
WordCount(), args);
System.exit(exitCode);
}
public int run(String[] args) throws Exception {
if (args.length != 2) {
System.out.printf("Usage: %s [generic options] <input
dir> <output dir>\n", getClass().getSimpleName());
return -1;
}
Job job = new Job(getConf());
job.setJarByClass(WordCount.class);
job.setJobName("Word Count");
FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setMapperClass(WordMapper.class);
job.setReducerClass(SumReducer.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
boolean success = job.waitForCompletion(true);
return success ? 0 : 1;
}
}
command line
-------------------
$ hadoop jar myjar.jar MyDriver -D mapred.reduce.tasks=10 myinputdir
myoutputdir
This is a better practise.
Happy Hadooping.
--
*Thanks & Regards*
Unmesha Sreeveni U.B
Hadoop, Bigdata Developer
Center for Cyber Security | Amrita Vishwa Vidyapeetham
http://www.unmeshasreeveni.blogspot.in/
Re: how to customize hadoop configuration for a job?
Posted by Mingjiang Shi <ms...@gopivotal.com>.
Hi Libo,
Could you post the exact command you run? Meanwhile, could you try to
remove the space between -D and the "property=value", i.e. use
-Dproperty=value instead of "-D property=value".
On Wed, Apr 2, 2014 at 9:48 AM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> According to the book "Hadoop; The Definitive Guide", it is possible to
> use "-D property=value" to
> override any default or site property in the configuration.
>
> I gave it shot and it is true. The property specified with "-D" is
> ignored.
>
> Then I put the property in an xml file and use "-conf xml_name" on the
> command line. But still I cannot
> override the property.
>
> The only way to override the default property is to get a Configuration
> reference in the code and set the property
> via the reference. But that is not convenient as I need to recompile the
> code each time I change the property.
> So what is the right way to customize the configuration for a job?
>
> Thanks,
>
> Libo
>
>
--
Cheers
-MJ
Re: how to customize hadoop configuration for a job?
Posted by Mingjiang Shi <ms...@gopivotal.com>.
Hi Libo,
Could you post the exact command you run? Meanwhile, could you try to
remove the space between -D and the "property=value", i.e. use
-Dproperty=value instead of "-D property=value".
On Wed, Apr 2, 2014 at 9:48 AM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> According to the book "Hadoop; The Definitive Guide", it is possible to
> use "-D property=value" to
> override any default or site property in the configuration.
>
> I gave it shot and it is true. The property specified with "-D" is
> ignored.
>
> Then I put the property in an xml file and use "-conf xml_name" on the
> command line. But still I cannot
> override the property.
>
> The only way to override the default property is to get a Configuration
> reference in the code and set the property
> via the reference. But that is not convenient as I need to recompile the
> code each time I change the property.
> So what is the right way to customize the configuration for a job?
>
> Thanks,
>
> Libo
>
>
--
Cheers
-MJ
Re: how to customize hadoop configuration for a job?
Posted by Mingjiang Shi <ms...@gopivotal.com>.
Hi Libo,
Could you post the exact command you run? Meanwhile, could you try to
remove the space between -D and the "property=value", i.e. use
-Dproperty=value instead of "-D property=value".
On Wed, Apr 2, 2014 at 9:48 AM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> According to the book "Hadoop; The Definitive Guide", it is possible to
> use "-D property=value" to
> override any default or site property in the configuration.
>
> I gave it shot and it is true. The property specified with "-D" is
> ignored.
>
> Then I put the property in an xml file and use "-conf xml_name" on the
> command line. But still I cannot
> override the property.
>
> The only way to override the default property is to get a Configuration
> reference in the code and set the property
> via the reference. But that is not convenient as I need to recompile the
> code each time I change the property.
> So what is the right way to customize the configuration for a job?
>
> Thanks,
>
> Libo
>
>
--
Cheers
-MJ
Re: how to customize hadoop configuration for a job?
Posted by Mingjiang Shi <ms...@gopivotal.com>.
Hi Libo,
Could you post the exact command you run? Meanwhile, could you try to
remove the space between -D and the "property=value", i.e. use
-Dproperty=value instead of "-D property=value".
On Wed, Apr 2, 2014 at 9:48 AM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> According to the book "Hadoop; The Definitive Guide", it is possible to
> use "-D property=value" to
> override any default or site property in the configuration.
>
> I gave it shot and it is true. The property specified with "-D" is
> ignored.
>
> Then I put the property in an xml file and use "-conf xml_name" on the
> command line. But still I cannot
> override the property.
>
> The only way to override the default property is to get a Configuration
> reference in the code and set the property
> via the reference. But that is not convenient as I need to recompile the
> code each time I change the property.
> So what is the right way to customize the configuration for a job?
>
> Thanks,
>
> Libo
>
>
--
Cheers
-MJ