You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by praveenesh kumar <pr...@gmail.com> on 2011/05/20 15:19:59 UTC
Why Only 1 Reducer is running ??
Hello everyone,
I am using wordcount application to test on my hadoop cluster of 5 nodes.
The file size is around 5 GB.
Its taking around 2 min - 40 sec for execution.
But when I am checking the JobTracker web portal, I am seeing only one
reducer is running. Why so ??
How can I change the code so that I will run multiple reducers also ??
Thanks,
Praveenesh
Re: Why Only 1 Reducer is running ??
Posted by praveenesh kumar <pr...@gmail.com>.
I am using the wordcount example that comes along with hadoop.
How can I configure it to make it use multiple reducers.
I guess mutiple reducers will make it run more fast .. Does it ??
On Fri, May 20, 2011 at 6:51 PM, James Seigel Tynt <ja...@tynt.com> wrote:
> The job could be designed to use one reducer
>
> On 2011-05-20, at 7:19 AM, praveenesh kumar <pr...@gmail.com> wrote:
>
> > Hello everyone,
> >
> > I am using wordcount application to test on my hadoop cluster of 5 nodes.
> > The file size is around 5 GB.
> > Its taking around 2 min - 40 sec for execution.
> > But when I am checking the JobTracker web portal, I am seeing only one
> > reducer is running. Why so ??
> > How can I change the code so that I will run multiple reducers also ??
> >
> > Thanks,
> > Praveenesh
>
Re: Why Only 1 Reducer is running ??
Posted by James Seigel Tynt <ja...@tynt.com>.
The job could be designed to use one reducer
On 2011-05-20, at 7:19 AM, praveenesh kumar <pr...@gmail.com> wrote:
> Hello everyone,
>
> I am using wordcount application to test on my hadoop cluster of 5 nodes.
> The file size is around 5 GB.
> Its taking around 2 min - 40 sec for execution.
> But when I am checking the JobTracker web portal, I am seeing only one
> reducer is running. Why so ??
> How can I change the code so that I will run multiple reducers also ??
>
> Thanks,
> Praveenesh
Re: Why Only 1 Reducer is running ??
Posted by praveenesh kumar <pr...@gmail.com>.
Okie I figured it out.. it was simple..
conf.setsetNumReduceTasks(10);
my mistake..
Anyhow when I am running 10 reducers for Wordcount problem.. I am seeing
only slight increase in the speed of the program... Why so ??
So more reducers do not gauranteee faster execution ??
How can we decide to use how many reducers to make our program run in the
best way possible ??
Thanks,
Praveenesh
On Mon, May 23, 2011 at 10:08 AM, praveenesh kumar <pr...@gmail.com>wrote:
> My program is a basic program like this :
>
> import java.io.IOException;
> import java.util.*;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.conf.*;
> import org.apache.hadoop.io.*;
> import org.apache.hadoop.mapred.*;
> import org.apache.hadoop.util.*;
>
> public class WordCount {
>
> public static class Map extends MapReduceBase implements
> Mapper<LongWritable, Text, Text, IntWritable> {
> private final static IntWritable one = new IntWritable(1);
> private Text word = new Text();
>
> public void map(LongWritable key, Text value, OutputCollector<Text,
> IntWritable> output, Reporter reporter) throws IOException {
> String line = value.toString();
> StringTokenizer tokenizer = new StringTokenizer(line);
> while (tokenizer.hasMoreTokens()) {
> word.set(tokenizer.nextToken());
> output.collect(word, one);
> }
> }
> }
>
> public static class Reduce extends MapReduceBase implements
> Reducer<Text, IntWritable, Text, IntWritable> {
> public void reduce(Text key, Iterator<IntWritable> values,
> OutputCollector<Text, IntWritable> output, Reporter reporter) throws
> IOException {
> int sum = 0;
> while (values.hasNext()) {
> sum += values.next().get();
> }
> output.collect(key, new IntWritable(sum));
> }
> }
>
> public static void main(String[] args) throws Exception {
> JobConf conf = new JobConf(WordCount.class);
> conf.setJobName("wordcount");
>
> conf.setOutputKeyClass(Text.class);
> conf.setOutputValueClass(IntWritable.class);
>
> conf.setMapperClass(Map.class);
> conf.setCombinerClass(Reduce.class);
> conf.setReducerClass(Reduce.class);
>
> conf.setInputFormat(TextInputFormat.class);
> conf.setOutputFormat(TextOutputFormat.class);
>
> FileInputFormat.setInputPaths(conf, new Path(args[0]));
> FileOutputFormat.setOutputPath(conf, new Path(args[1]));
> Job.setNumReduceTasks(10);
> JobClient.runJob(conf);
> }
> }
>
>
> How to use Job.setNumReduceTasks(INT) function here.. I am not using any
> Job class object here.
>
> Thanks.
> Praveenesh
>
>
> On Fri, May 20, 2011 at 7:07 PM, Evert Lammerts <Ev...@sara.nl>wrote:
>
>> Hi Praveenesh,
>>
>> * You can set the maximum amount of reducers per node in your
>> mapred-site.xml using mapred.tasktracker.reduce.tasks.maximum (default set
>> to 2).
>> * You can set the default number of reduce tasks with mapred.reduce.tasks
>> (default set to 1 - this causes your single reducer).
>> * Your job can try to override this setting by calling
>> Job.setNumReduceTasks(INT) (
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#setNumReduceTasks(int)
>> ).
>>
>> Cheers,
>> Evert
>>
>>
>> > -----Original Message-----
>> > From: modemide [mailto:modemide@gmail.com]
>> > Sent: vrijdag 20 mei 2011 15:26
>> > To: common-user@hadoop.apache.org
>> > Subject: Re: Why Only 1 Reducer is running ??
>> >
>> > what does your mapred-site.xml file say?
>> >
>> > I've used wordcount and had close to 12 reduces running on a 6
>> > datanode cluster on a 3 GB file.
>> >
>> >
>> > I have a configuration in there which says:
>> > mapred.reduce.tasks = 12
>> >
>> > The reason I chose 12 was because it was recommended that I choose 2x
>> > number of tasktrackers.
>> >
>> >
>> >
>> >
>> >
>> > On 5/20/11, praveenesh kumar <pr...@gmail.com> wrote:
>> > > Hello everyone,
>> > >
>> > > I am using wordcount application to test on my hadoop cluster of 5
>> > nodes.
>> > > The file size is around 5 GB.
>> > > Its taking around 2 min - 40 sec for execution.
>> > > But when I am checking the JobTracker web portal, I am seeing only
>> > one
>> > > reducer is running. Why so ??
>> > > How can I change the code so that I will run multiple reducers also
>> > ??
>> > >
>> > > Thanks,
>> > > Praveenesh
>> > >
>>
>
>
Re: Why Only 1 Reducer is running ??
Posted by praveenesh kumar <pr...@gmail.com>.
My program is a basic program like this :
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
public class WordCount {
public static class Map extends MapReduceBase implements
Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, OutputCollector<Text,
IntWritable> output, Reporter reporter) throws IOException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
output.collect(word, one);
}
}
}
public static class Reduce extends MapReduceBase implements
Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output, Reporter reporter) throws
IOException {
int sum = 0;
while (values.hasNext()) {
sum += values.next().get();
}
output.collect(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
JobConf conf = new JobConf(WordCount.class);
conf.setJobName("wordcount");
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(Map.class);
conf.setCombinerClass(Reduce.class);
conf.setReducerClass(Reduce.class);
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
FileInputFormat.setInputPaths(conf, new Path(args[0]));
FileOutputFormat.setOutputPath(conf, new Path(args[1]));
Job.Job.setNumReduceTasks(10);
JobClient.runJob(conf);
}
}
How to use Job.setNumReduceTasks(INT) function here.. I am not using any Job
class object here.
Thanks.
Praveenesh
On Fri, May 20, 2011 at 7:07 PM, Evert Lammerts <Ev...@sara.nl>wrote:
> Hi Praveenesh,
>
> * You can set the maximum amount of reducers per node in your
> mapred-site.xml using mapred.tasktracker.reduce.tasks.maximum (default set
> to 2).
> * You can set the default number of reduce tasks with mapred.reduce.tasks
> (default set to 1 - this causes your single reducer).
> * Your job can try to override this setting by calling
> Job.setNumReduceTasks(INT) (
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#setNumReduceTasks(int)
> ).
>
> Cheers,
> Evert
>
>
> > -----Original Message-----
> > From: modemide [mailto:modemide@gmail.com]
> > Sent: vrijdag 20 mei 2011 15:26
> > To: common-user@hadoop.apache.org
> > Subject: Re: Why Only 1 Reducer is running ??
> >
> > what does your mapred-site.xml file say?
> >
> > I've used wordcount and had close to 12 reduces running on a 6
> > datanode cluster on a 3 GB file.
> >
> >
> > I have a configuration in there which says:
> > mapred.reduce.tasks = 12
> >
> > The reason I chose 12 was because it was recommended that I choose 2x
> > number of tasktrackers.
> >
> >
> >
> >
> >
> > On 5/20/11, praveenesh kumar <pr...@gmail.com> wrote:
> > > Hello everyone,
> > >
> > > I am using wordcount application to test on my hadoop cluster of 5
> > nodes.
> > > The file size is around 5 GB.
> > > Its taking around 2 min - 40 sec for execution.
> > > But when I am checking the JobTracker web portal, I am seeing only
> > one
> > > reducer is running. Why so ??
> > > How can I change the code so that I will run multiple reducers also
> > ??
> > >
> > > Thanks,
> > > Praveenesh
> > >
>
RE: Why Only 1 Reducer is running ??
Posted by Evert Lammerts <Ev...@sara.nl>.
Hi Praveenesh,
* You can set the maximum amount of reducers per node in your mapred-site.xml using mapred.tasktracker.reduce.tasks.maximum (default set to 2).
* You can set the default number of reduce tasks with mapred.reduce.tasks (default set to 1 - this causes your single reducer).
* Your job can try to override this setting by calling Job.setNumReduceTasks(INT) (http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#setNumReduceTasks(int)).
Cheers,
Evert
> -----Original Message-----
> From: modemide [mailto:modemide@gmail.com]
> Sent: vrijdag 20 mei 2011 15:26
> To: common-user@hadoop.apache.org
> Subject: Re: Why Only 1 Reducer is running ??
>
> what does your mapred-site.xml file say?
>
> I've used wordcount and had close to 12 reduces running on a 6
> datanode cluster on a 3 GB file.
>
>
> I have a configuration in there which says:
> mapred.reduce.tasks = 12
>
> The reason I chose 12 was because it was recommended that I choose 2x
> number of tasktrackers.
>
>
>
>
>
> On 5/20/11, praveenesh kumar <pr...@gmail.com> wrote:
> > Hello everyone,
> >
> > I am using wordcount application to test on my hadoop cluster of 5
> nodes.
> > The file size is around 5 GB.
> > Its taking around 2 min - 40 sec for execution.
> > But when I am checking the JobTracker web portal, I am seeing only
> one
> > reducer is running. Why so ??
> > How can I change the code so that I will run multiple reducers also
> ??
> >
> > Thanks,
> > Praveenesh
> >
Re: Why Only 1 Reducer is running ??
Posted by praveenesh kumar <pr...@gmail.com>.
Hi,
My mapred-site.xml is pretty simple.
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>ub13:54311</value>
<description>The host and port that the MapReduce job tracker runs at. If
"local", then jobs are run in-process as a single map and reduce task.
</description>
</property>
</configuration>
where I should put the settings that you are saying ??
On Fri, May 20, 2011 at 6:55 PM, modemide <mo...@gmail.com> wrote:
> what does your mapred-site.xml file say?
>
> I've used wordcount and had close to 12 reduces running on a 6
> datanode cluster on a 3 GB file.
>
>
> I have a configuration in there which says:
> mapred.reduce.tasks = 12
>
> The reason I chose 12 was because it was recommended that I choose 2x
> number of tasktrackers.
>
>
>
>
>
> On 5/20/11, praveenesh kumar <pr...@gmail.com> wrote:
> > Hello everyone,
> >
> > I am using wordcount application to test on my hadoop cluster of 5 nodes.
> > The file size is around 5 GB.
> > Its taking around 2 min - 40 sec for execution.
> > But when I am checking the JobTracker web portal, I am seeing only one
> > reducer is running. Why so ??
> > How can I change the code so that I will run multiple reducers also ??
> >
> > Thanks,
> > Praveenesh
> >
>
Re: Why Only 1 Reducer is running ??
Posted by modemide <mo...@gmail.com>.
what does your mapred-site.xml file say?
I've used wordcount and had close to 12 reduces running on a 6
datanode cluster on a 3 GB file.
I have a configuration in there which says:
mapred.reduce.tasks = 12
The reason I chose 12 was because it was recommended that I choose 2x
number of tasktrackers.
On 5/20/11, praveenesh kumar <pr...@gmail.com> wrote:
> Hello everyone,
>
> I am using wordcount application to test on my hadoop cluster of 5 nodes.
> The file size is around 5 GB.
> Its taking around 2 min - 40 sec for execution.
> But when I am checking the JobTracker web portal, I am seeing only one
> reducer is running. Why so ??
> How can I change the code so that I will run multiple reducers also ??
>
> Thanks,
> Praveenesh
>