You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Ravikant Dindokar <ra...@gmail.com> on 2015/06/26 16:02:52 UTC
Only one reducer working
Hi Hadoop User,
I am processing 23 GB file on 21 nodes.
I have tried both options :
mapreduce.job.reduces=50
&
mapred.tasktracker.reduce.tasks.maximum=5
in mapred-site.xml
but still only one reducer is running.
Any configuration setting still to be corrected?
Thanks
Ravikant
Re: Only one reducer working
Posted by Ravikant Dindokar <ra...@gmail.com>.
set the value directly in code. Given a JobConf instance job, call
job.setNumReduceTasks(100);
This worked for me
Thanks
On Fri, Jun 26, 2015 at 9:07 PM, Ravikant Dindokar <ra...@gmail.com>
wrote:
> text file with tab separated values
>
> On Fri, Jun 26, 2015 at 7:35 PM, Krishna Kalyan <kr...@gmail.com>
> wrote:
>
>> What is the file format?.
>>
>> Thanks,
>> Krishna
>> On 26 Jun 2015 7:33 pm, "Ravikant Dindokar" <ra...@gmail.com>
>> wrote:
>>
>>> Hi Hadoop User,
>>>
>>> I am processing 23 GB file on 21 nodes.
>>>
>>> I have tried both options :
>>>
>>> mapreduce.job.reduces=50
>>>
>>> &
>>>
>>> mapred.tasktracker.reduce.tasks.maximum=5
>>> in mapred-site.xml
>>>
>>> but still only one reducer is running.
>>>
>>> Any configuration setting still to be corrected?
>>>
>>> Thanks
>>> Ravikant
>>>
>>
>
Re: Only one reducer working
Posted by Ravikant Dindokar <ra...@gmail.com>.
set the value directly in code. Given a JobConf instance job, call
job.setNumReduceTasks(100);
This worked for me
Thanks
On Fri, Jun 26, 2015 at 9:07 PM, Ravikant Dindokar <ra...@gmail.com>
wrote:
> text file with tab separated values
>
> On Fri, Jun 26, 2015 at 7:35 PM, Krishna Kalyan <kr...@gmail.com>
> wrote:
>
>> What is the file format?.
>>
>> Thanks,
>> Krishna
>> On 26 Jun 2015 7:33 pm, "Ravikant Dindokar" <ra...@gmail.com>
>> wrote:
>>
>>> Hi Hadoop User,
>>>
>>> I am processing 23 GB file on 21 nodes.
>>>
>>> I have tried both options :
>>>
>>> mapreduce.job.reduces=50
>>>
>>> &
>>>
>>> mapred.tasktracker.reduce.tasks.maximum=5
>>> in mapred-site.xml
>>>
>>> but still only one reducer is running.
>>>
>>> Any configuration setting still to be corrected?
>>>
>>> Thanks
>>> Ravikant
>>>
>>
>
Re: Only one reducer working
Posted by Ravikant Dindokar <ra...@gmail.com>.
set the value directly in code. Given a JobConf instance job, call
job.setNumReduceTasks(100);
This worked for me
Thanks
On Fri, Jun 26, 2015 at 9:07 PM, Ravikant Dindokar <ra...@gmail.com>
wrote:
> text file with tab separated values
>
> On Fri, Jun 26, 2015 at 7:35 PM, Krishna Kalyan <kr...@gmail.com>
> wrote:
>
>> What is the file format?.
>>
>> Thanks,
>> Krishna
>> On 26 Jun 2015 7:33 pm, "Ravikant Dindokar" <ra...@gmail.com>
>> wrote:
>>
>>> Hi Hadoop User,
>>>
>>> I am processing 23 GB file on 21 nodes.
>>>
>>> I have tried both options :
>>>
>>> mapreduce.job.reduces=50
>>>
>>> &
>>>
>>> mapred.tasktracker.reduce.tasks.maximum=5
>>> in mapred-site.xml
>>>
>>> but still only one reducer is running.
>>>
>>> Any configuration setting still to be corrected?
>>>
>>> Thanks
>>> Ravikant
>>>
>>
>
Re: Only one reducer working
Posted by Ravikant Dindokar <ra...@gmail.com>.
set the value directly in code. Given a JobConf instance job, call
job.setNumReduceTasks(100);
This worked for me
Thanks
On Fri, Jun 26, 2015 at 9:07 PM, Ravikant Dindokar <ra...@gmail.com>
wrote:
> text file with tab separated values
>
> On Fri, Jun 26, 2015 at 7:35 PM, Krishna Kalyan <kr...@gmail.com>
> wrote:
>
>> What is the file format?.
>>
>> Thanks,
>> Krishna
>> On 26 Jun 2015 7:33 pm, "Ravikant Dindokar" <ra...@gmail.com>
>> wrote:
>>
>>> Hi Hadoop User,
>>>
>>> I am processing 23 GB file on 21 nodes.
>>>
>>> I have tried both options :
>>>
>>> mapreduce.job.reduces=50
>>>
>>> &
>>>
>>> mapred.tasktracker.reduce.tasks.maximum=5
>>> in mapred-site.xml
>>>
>>> but still only one reducer is running.
>>>
>>> Any configuration setting still to be corrected?
>>>
>>> Thanks
>>> Ravikant
>>>
>>
>
Re: Only one reducer working
Posted by Ravikant Dindokar <ra...@gmail.com>.
text file with tab separated values
On Fri, Jun 26, 2015 at 7:35 PM, Krishna Kalyan <kr...@gmail.com>
wrote:
> What is the file format?.
>
> Thanks,
> Krishna
> On 26 Jun 2015 7:33 pm, "Ravikant Dindokar" <ra...@gmail.com>
> wrote:
>
>> Hi Hadoop User,
>>
>> I am processing 23 GB file on 21 nodes.
>>
>> I have tried both options :
>>
>> mapreduce.job.reduces=50
>>
>> &
>>
>> mapred.tasktracker.reduce.tasks.maximum=5
>> in mapred-site.xml
>>
>> but still only one reducer is running.
>>
>> Any configuration setting still to be corrected?
>>
>> Thanks
>> Ravikant
>>
>
Re: Only one reducer working
Posted by Ravikant Dindokar <ra...@gmail.com>.
text file with tab separated values
On Fri, Jun 26, 2015 at 7:35 PM, Krishna Kalyan <kr...@gmail.com>
wrote:
> What is the file format?.
>
> Thanks,
> Krishna
> On 26 Jun 2015 7:33 pm, "Ravikant Dindokar" <ra...@gmail.com>
> wrote:
>
>> Hi Hadoop User,
>>
>> I am processing 23 GB file on 21 nodes.
>>
>> I have tried both options :
>>
>> mapreduce.job.reduces=50
>>
>> &
>>
>> mapred.tasktracker.reduce.tasks.maximum=5
>> in mapred-site.xml
>>
>> but still only one reducer is running.
>>
>> Any configuration setting still to be corrected?
>>
>> Thanks
>> Ravikant
>>
>
Re: Only one reducer working
Posted by Ravikant Dindokar <ra...@gmail.com>.
text file with tab separated values
On Fri, Jun 26, 2015 at 7:35 PM, Krishna Kalyan <kr...@gmail.com>
wrote:
> What is the file format?.
>
> Thanks,
> Krishna
> On 26 Jun 2015 7:33 pm, "Ravikant Dindokar" <ra...@gmail.com>
> wrote:
>
>> Hi Hadoop User,
>>
>> I am processing 23 GB file on 21 nodes.
>>
>> I have tried both options :
>>
>> mapreduce.job.reduces=50
>>
>> &
>>
>> mapred.tasktracker.reduce.tasks.maximum=5
>> in mapred-site.xml
>>
>> but still only one reducer is running.
>>
>> Any configuration setting still to be corrected?
>>
>> Thanks
>> Ravikant
>>
>
Re: Only one reducer working
Posted by Ravikant Dindokar <ra...@gmail.com>.
text file with tab separated values
On Fri, Jun 26, 2015 at 7:35 PM, Krishna Kalyan <kr...@gmail.com>
wrote:
> What is the file format?.
>
> Thanks,
> Krishna
> On 26 Jun 2015 7:33 pm, "Ravikant Dindokar" <ra...@gmail.com>
> wrote:
>
>> Hi Hadoop User,
>>
>> I am processing 23 GB file on 21 nodes.
>>
>> I have tried both options :
>>
>> mapreduce.job.reduces=50
>>
>> &
>>
>> mapred.tasktracker.reduce.tasks.maximum=5
>> in mapred-site.xml
>>
>> but still only one reducer is running.
>>
>> Any configuration setting still to be corrected?
>>
>> Thanks
>> Ravikant
>>
>
Re: Only one reducer working
Posted by Krishna Kalyan <kr...@gmail.com>.
What is the file format?.
Thanks,
Krishna
On 26 Jun 2015 7:33 pm, "Ravikant Dindokar" <ra...@gmail.com> wrote:
> Hi Hadoop User,
>
> I am processing 23 GB file on 21 nodes.
>
> I have tried both options :
>
> mapreduce.job.reduces=50
>
> &
>
> mapred.tasktracker.reduce.tasks.maximum=5
> in mapred-site.xml
>
> but still only one reducer is running.
>
> Any configuration setting still to be corrected?
>
> Thanks
> Ravikant
>
Re: Only one reducer working
Posted by Krishna Kalyan <kr...@gmail.com>.
What is the file format?.
Thanks,
Krishna
On 26 Jun 2015 7:33 pm, "Ravikant Dindokar" <ra...@gmail.com> wrote:
> Hi Hadoop User,
>
> I am processing 23 GB file on 21 nodes.
>
> I have tried both options :
>
> mapreduce.job.reduces=50
>
> &
>
> mapred.tasktracker.reduce.tasks.maximum=5
> in mapred-site.xml
>
> but still only one reducer is running.
>
> Any configuration setting still to be corrected?
>
> Thanks
> Ravikant
>
Re: Only one reducer working
Posted by Krishna Kalyan <kr...@gmail.com>.
What is the file format?.
Thanks,
Krishna
On 26 Jun 2015 7:33 pm, "Ravikant Dindokar" <ra...@gmail.com> wrote:
> Hi Hadoop User,
>
> I am processing 23 GB file on 21 nodes.
>
> I have tried both options :
>
> mapreduce.job.reduces=50
>
> &
>
> mapred.tasktracker.reduce.tasks.maximum=5
> in mapred-site.xml
>
> but still only one reducer is running.
>
> Any configuration setting still to be corrected?
>
> Thanks
> Ravikant
>
Re: Only one reducer working
Posted by Krishna Kalyan <kr...@gmail.com>.
What is the file format?.
Thanks,
Krishna
On 26 Jun 2015 7:33 pm, "Ravikant Dindokar" <ra...@gmail.com> wrote:
> Hi Hadoop User,
>
> I am processing 23 GB file on 21 nodes.
>
> I have tried both options :
>
> mapreduce.job.reduces=50
>
> &
>
> mapred.tasktracker.reduce.tasks.maximum=5
> in mapred-site.xml
>
> but still only one reducer is running.
>
> Any configuration setting still to be corrected?
>
> Thanks
> Ravikant
>