You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by xeon <xe...@gmail.com> on 2013/09/26 00:01:00 UTC
set the number of reduce tasks in the wordcount by command line
is it possible to set the number of reduce tasks in the wordcount
example when I launch the job by command line?
Thanks
Re: set the number of reduce tasks in the wordcount by command line
Posted by Shahab Yunus <sh...@gmail.com>.
See an example here:
http://www.crobak.org/2011/12/getting-started-with-apache-hadoop-0-23-0/
For the 'pi' program. Basically you will specify it through the command
line interface.
On Wed, Sep 25, 2013 at 6:17 PM, xeon <xe...@gmail.com> wrote:
> In yarn 2.0.5, where I set this?
>
> On 09/25/2013 11:16 PM, Shahab Yunus wrote:
>
> Have you tried setting *mapred.reduce.tasks *property?
>
> Regards,
> Shahab
>
>
> On Wed, Sep 25, 2013 at 6:01 PM, xeon <xe...@gmail.com> wrote:
>
>> is it possible to set the number of reduce tasks in the wordcount example
>> when I launch the job by command line?
>>
>> Thanks
>>
>
>
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by Shahab Yunus <sh...@gmail.com>.
Yeah, thanks Krishna for pointing out Yarn specific property name.
Regards,
Shahab
On Wed, Sep 25, 2013 at 6:54 PM, Krishna Pisupat
<kr...@gmail.com>wrote:
> You can invoke the setNumReduceTasks on the Job object that you use to run
> the MR job.
>
>
> http://hadoop.apache.org/docs/r2.0.6-alpha/api/org/apache/hadoop/mapreduce/Job.html#setNumReduceTasks(int)
>
> Or else you can set the property mapreduce.job.reduces in mapred-site.xml
>
>
> mapreduce.job.reduces1The default number of reduce tasks per job.
> Typically set to 99% of the cluster's reduce capacity, so that if a node
> fails the reduces can still be executed in a single wave. Ignored when
> mapreduce.jobtracker.address is "local".
>
>
>
>
> On Sep 25, 2013, at 3:17 PM, xeon <xe...@gmail.com> wrote:
>
> In yarn 2.0.5, where I set this?
>
> On 09/25/2013 11:16 PM, Shahab Yunus wrote:
>
> Have you tried setting *mapred.reduce.tasks *property?
>
> Regards,
> Shahab
>
>
> On Wed, Sep 25, 2013 at 6:01 PM, xeon <xe...@gmail.com> wrote:
>
>> is it possible to set the number of reduce tasks in the wordcount example
>> when I launch the job by command line?
>>
>> Thanks
>>
>
>
>
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by Shahab Yunus <sh...@gmail.com>.
Yeah, thanks Krishna for pointing out Yarn specific property name.
Regards,
Shahab
On Wed, Sep 25, 2013 at 6:54 PM, Krishna Pisupat
<kr...@gmail.com>wrote:
> You can invoke the setNumReduceTasks on the Job object that you use to run
> the MR job.
>
>
> http://hadoop.apache.org/docs/r2.0.6-alpha/api/org/apache/hadoop/mapreduce/Job.html#setNumReduceTasks(int)
>
> Or else you can set the property mapreduce.job.reduces in mapred-site.xml
>
>
> mapreduce.job.reduces1The default number of reduce tasks per job.
> Typically set to 99% of the cluster's reduce capacity, so that if a node
> fails the reduces can still be executed in a single wave. Ignored when
> mapreduce.jobtracker.address is "local".
>
>
>
>
> On Sep 25, 2013, at 3:17 PM, xeon <xe...@gmail.com> wrote:
>
> In yarn 2.0.5, where I set this?
>
> On 09/25/2013 11:16 PM, Shahab Yunus wrote:
>
> Have you tried setting *mapred.reduce.tasks *property?
>
> Regards,
> Shahab
>
>
> On Wed, Sep 25, 2013 at 6:01 PM, xeon <xe...@gmail.com> wrote:
>
>> is it possible to set the number of reduce tasks in the wordcount example
>> when I launch the job by command line?
>>
>> Thanks
>>
>
>
>
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by Shahab Yunus <sh...@gmail.com>.
Yeah, thanks Krishna for pointing out Yarn specific property name.
Regards,
Shahab
On Wed, Sep 25, 2013 at 6:54 PM, Krishna Pisupat
<kr...@gmail.com>wrote:
> You can invoke the setNumReduceTasks on the Job object that you use to run
> the MR job.
>
>
> http://hadoop.apache.org/docs/r2.0.6-alpha/api/org/apache/hadoop/mapreduce/Job.html#setNumReduceTasks(int)
>
> Or else you can set the property mapreduce.job.reduces in mapred-site.xml
>
>
> mapreduce.job.reduces1The default number of reduce tasks per job.
> Typically set to 99% of the cluster's reduce capacity, so that if a node
> fails the reduces can still be executed in a single wave. Ignored when
> mapreduce.jobtracker.address is "local".
>
>
>
>
> On Sep 25, 2013, at 3:17 PM, xeon <xe...@gmail.com> wrote:
>
> In yarn 2.0.5, where I set this?
>
> On 09/25/2013 11:16 PM, Shahab Yunus wrote:
>
> Have you tried setting *mapred.reduce.tasks *property?
>
> Regards,
> Shahab
>
>
> On Wed, Sep 25, 2013 at 6:01 PM, xeon <xe...@gmail.com> wrote:
>
>> is it possible to set the number of reduce tasks in the wordcount example
>> when I launch the job by command line?
>>
>> Thanks
>>
>
>
>
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by Shahab Yunus <sh...@gmail.com>.
Yeah, thanks Krishna for pointing out Yarn specific property name.
Regards,
Shahab
On Wed, Sep 25, 2013 at 6:54 PM, Krishna Pisupat
<kr...@gmail.com>wrote:
> You can invoke the setNumReduceTasks on the Job object that you use to run
> the MR job.
>
>
> http://hadoop.apache.org/docs/r2.0.6-alpha/api/org/apache/hadoop/mapreduce/Job.html#setNumReduceTasks(int)
>
> Or else you can set the property mapreduce.job.reduces in mapred-site.xml
>
>
> mapreduce.job.reduces1The default number of reduce tasks per job.
> Typically set to 99% of the cluster's reduce capacity, so that if a node
> fails the reduces can still be executed in a single wave. Ignored when
> mapreduce.jobtracker.address is "local".
>
>
>
>
> On Sep 25, 2013, at 3:17 PM, xeon <xe...@gmail.com> wrote:
>
> In yarn 2.0.5, where I set this?
>
> On 09/25/2013 11:16 PM, Shahab Yunus wrote:
>
> Have you tried setting *mapred.reduce.tasks *property?
>
> Regards,
> Shahab
>
>
> On Wed, Sep 25, 2013 at 6:01 PM, xeon <xe...@gmail.com> wrote:
>
>> is it possible to set the number of reduce tasks in the wordcount example
>> when I launch the job by command line?
>>
>> Thanks
>>
>
>
>
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by Krishna Pisupat <kr...@gmail.com>.
You can invoke the setNumReduceTasks on the Job object that you use to run the MR job.
http://hadoop.apache.org/docs/r2.0.6-alpha/api/org/apache/hadoop/mapreduce/Job.html#setNumReduceTasks(int)
Or else you can set the property mapreduce.job.reduces in mapred-site.xml
mapreduce.job.reduces 1 The default number of reduce tasks per job. Typically set to 99% of the cluster's reduce capacity, so that if a node fails the reduces can still be executed in a single wave. Ignored when mapreduce.jobtracker.address is "local".
On Sep 25, 2013, at 3:17 PM, xeon <xe...@gmail.com> wrote:
> In yarn 2.0.5, where I set this?
>
> On 09/25/2013 11:16 PM, Shahab Yunus wrote:
>> Have you tried setting mapred.reduce.tasks property?
>>
>> Regards,
>> Shahab
>>
>>
>> On Wed, Sep 25, 2013 at 6:01 PM, xeon <xe...@gmail.com> wrote:
>> is it possible to set the number of reduce tasks in the wordcount example when I launch the job by command line?
>>
>> Thanks
>>
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by Krishna Pisupat <kr...@gmail.com>.
You can invoke the setNumReduceTasks on the Job object that you use to run the MR job.
http://hadoop.apache.org/docs/r2.0.6-alpha/api/org/apache/hadoop/mapreduce/Job.html#setNumReduceTasks(int)
Or else you can set the property mapreduce.job.reduces in mapred-site.xml
mapreduce.job.reduces 1 The default number of reduce tasks per job. Typically set to 99% of the cluster's reduce capacity, so that if a node fails the reduces can still be executed in a single wave. Ignored when mapreduce.jobtracker.address is "local".
On Sep 25, 2013, at 3:17 PM, xeon <xe...@gmail.com> wrote:
> In yarn 2.0.5, where I set this?
>
> On 09/25/2013 11:16 PM, Shahab Yunus wrote:
>> Have you tried setting mapred.reduce.tasks property?
>>
>> Regards,
>> Shahab
>>
>>
>> On Wed, Sep 25, 2013 at 6:01 PM, xeon <xe...@gmail.com> wrote:
>> is it possible to set the number of reduce tasks in the wordcount example when I launch the job by command line?
>>
>> Thanks
>>
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by Shahab Yunus <sh...@gmail.com>.
See an example here:
http://www.crobak.org/2011/12/getting-started-with-apache-hadoop-0-23-0/
For the 'pi' program. Basically you will specify it through the command
line interface.
On Wed, Sep 25, 2013 at 6:17 PM, xeon <xe...@gmail.com> wrote:
> In yarn 2.0.5, where I set this?
>
> On 09/25/2013 11:16 PM, Shahab Yunus wrote:
>
> Have you tried setting *mapred.reduce.tasks *property?
>
> Regards,
> Shahab
>
>
> On Wed, Sep 25, 2013 at 6:01 PM, xeon <xe...@gmail.com> wrote:
>
>> is it possible to set the number of reduce tasks in the wordcount example
>> when I launch the job by command line?
>>
>> Thanks
>>
>
>
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by Krishna Pisupat <kr...@gmail.com>.
You can invoke the setNumReduceTasks on the Job object that you use to run the MR job.
http://hadoop.apache.org/docs/r2.0.6-alpha/api/org/apache/hadoop/mapreduce/Job.html#setNumReduceTasks(int)
Or else you can set the property mapreduce.job.reduces in mapred-site.xml
mapreduce.job.reduces 1 The default number of reduce tasks per job. Typically set to 99% of the cluster's reduce capacity, so that if a node fails the reduces can still be executed in a single wave. Ignored when mapreduce.jobtracker.address is "local".
On Sep 25, 2013, at 3:17 PM, xeon <xe...@gmail.com> wrote:
> In yarn 2.0.5, where I set this?
>
> On 09/25/2013 11:16 PM, Shahab Yunus wrote:
>> Have you tried setting mapred.reduce.tasks property?
>>
>> Regards,
>> Shahab
>>
>>
>> On Wed, Sep 25, 2013 at 6:01 PM, xeon <xe...@gmail.com> wrote:
>> is it possible to set the number of reduce tasks in the wordcount example when I launch the job by command line?
>>
>> Thanks
>>
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by Krishna Pisupat <kr...@gmail.com>.
You can invoke the setNumReduceTasks on the Job object that you use to run the MR job.
http://hadoop.apache.org/docs/r2.0.6-alpha/api/org/apache/hadoop/mapreduce/Job.html#setNumReduceTasks(int)
Or else you can set the property mapreduce.job.reduces in mapred-site.xml
mapreduce.job.reduces 1 The default number of reduce tasks per job. Typically set to 99% of the cluster's reduce capacity, so that if a node fails the reduces can still be executed in a single wave. Ignored when mapreduce.jobtracker.address is "local".
On Sep 25, 2013, at 3:17 PM, xeon <xe...@gmail.com> wrote:
> In yarn 2.0.5, where I set this?
>
> On 09/25/2013 11:16 PM, Shahab Yunus wrote:
>> Have you tried setting mapred.reduce.tasks property?
>>
>> Regards,
>> Shahab
>>
>>
>> On Wed, Sep 25, 2013 at 6:01 PM, xeon <xe...@gmail.com> wrote:
>> is it possible to set the number of reduce tasks in the wordcount example when I launch the job by command line?
>>
>> Thanks
>>
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by Shahab Yunus <sh...@gmail.com>.
See an example here:
http://www.crobak.org/2011/12/getting-started-with-apache-hadoop-0-23-0/
For the 'pi' program. Basically you will specify it through the command
line interface.
On Wed, Sep 25, 2013 at 6:17 PM, xeon <xe...@gmail.com> wrote:
> In yarn 2.0.5, where I set this?
>
> On 09/25/2013 11:16 PM, Shahab Yunus wrote:
>
> Have you tried setting *mapred.reduce.tasks *property?
>
> Regards,
> Shahab
>
>
> On Wed, Sep 25, 2013 at 6:01 PM, xeon <xe...@gmail.com> wrote:
>
>> is it possible to set the number of reduce tasks in the wordcount example
>> when I launch the job by command line?
>>
>> Thanks
>>
>
>
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by Shahab Yunus <sh...@gmail.com>.
See an example here:
http://www.crobak.org/2011/12/getting-started-with-apache-hadoop-0-23-0/
For the 'pi' program. Basically you will specify it through the command
line interface.
On Wed, Sep 25, 2013 at 6:17 PM, xeon <xe...@gmail.com> wrote:
> In yarn 2.0.5, where I set this?
>
> On 09/25/2013 11:16 PM, Shahab Yunus wrote:
>
> Have you tried setting *mapred.reduce.tasks *property?
>
> Regards,
> Shahab
>
>
> On Wed, Sep 25, 2013 at 6:01 PM, xeon <xe...@gmail.com> wrote:
>
>> is it possible to set the number of reduce tasks in the wordcount example
>> when I launch the job by command line?
>>
>> Thanks
>>
>
>
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by xeon <xe...@gmail.com>.
In yarn 2.0.5, where I set this?
On 09/25/2013 11:16 PM, Shahab Yunus wrote:
> Have you tried setting *mapred.reduce.tasks *property?
>
> Regards,
> Shahab
>
>
> On Wed, Sep 25, 2013 at 6:01 PM, xeon <xeonmailinglist@gmail.com
> <ma...@gmail.com>> wrote:
>
> is it possible to set the number of reduce tasks in the wordcount
> example when I launch the job by command line?
>
> Thanks
>
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by xeon <xe...@gmail.com>.
In yarn 2.0.5, where I set this?
On 09/25/2013 11:16 PM, Shahab Yunus wrote:
> Have you tried setting *mapred.reduce.tasks *property?
>
> Regards,
> Shahab
>
>
> On Wed, Sep 25, 2013 at 6:01 PM, xeon <xeonmailinglist@gmail.com
> <ma...@gmail.com>> wrote:
>
> is it possible to set the number of reduce tasks in the wordcount
> example when I launch the job by command line?
>
> Thanks
>
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by xeon <xe...@gmail.com>.
In yarn 2.0.5, where I set this?
On 09/25/2013 11:16 PM, Shahab Yunus wrote:
> Have you tried setting *mapred.reduce.tasks *property?
>
> Regards,
> Shahab
>
>
> On Wed, Sep 25, 2013 at 6:01 PM, xeon <xeonmailinglist@gmail.com
> <ma...@gmail.com>> wrote:
>
> is it possible to set the number of reduce tasks in the wordcount
> example when I launch the job by command line?
>
> Thanks
>
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by xeon <xe...@gmail.com>.
In yarn 2.0.5, where I set this?
On 09/25/2013 11:16 PM, Shahab Yunus wrote:
> Have you tried setting *mapred.reduce.tasks *property?
>
> Regards,
> Shahab
>
>
> On Wed, Sep 25, 2013 at 6:01 PM, xeon <xeonmailinglist@gmail.com
> <ma...@gmail.com>> wrote:
>
> is it possible to set the number of reduce tasks in the wordcount
> example when I launch the job by command line?
>
> Thanks
>
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by Shahab Yunus <sh...@gmail.com>.
Have you tried setting *mapred.reduce.tasks *property?
Regards,
Shahab
On Wed, Sep 25, 2013 at 6:01 PM, xeon <xe...@gmail.com> wrote:
> is it possible to set the number of reduce tasks in the wordcount example
> when I launch the job by command line?
>
> Thanks
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by Shahab Yunus <sh...@gmail.com>.
Have you tried setting *mapred.reduce.tasks *property?
Regards,
Shahab
On Wed, Sep 25, 2013 at 6:01 PM, xeon <xe...@gmail.com> wrote:
> is it possible to set the number of reduce tasks in the wordcount example
> when I launch the job by command line?
>
> Thanks
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by Shahab Yunus <sh...@gmail.com>.
Have you tried setting *mapred.reduce.tasks *property?
Regards,
Shahab
On Wed, Sep 25, 2013 at 6:01 PM, xeon <xe...@gmail.com> wrote:
> is it possible to set the number of reduce tasks in the wordcount example
> when I launch the job by command line?
>
> Thanks
>
Re: set the number of reduce tasks in the wordcount by command line
Posted by Shahab Yunus <sh...@gmail.com>.
Have you tried setting *mapred.reduce.tasks *property?
Regards,
Shahab
On Wed, Sep 25, 2013 at 6:01 PM, xeon <xe...@gmail.com> wrote:
> is it possible to set the number of reduce tasks in the wordcount example
> when I launch the job by command line?
>
> Thanks
>