You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Pedro Costa <ps...@gmail.com> on 2011/05/23 18:45:31 UTC
How set number of map and reduce can run simultaneously
Hi,
I'm running hadoop map-reduce in a cluster with 10 machines. I would
like to set in the configuration that each tasktracker can run 8 map
tasks simultaneously and 4 reduce tasks simultaneously. Which
parameters should I configure?
Thanks,
PSC
Re: How set number of map and reduce can run simultaneously
Posted by Pedro Costa <ps...@gmail.com>.
I found the solution. The problem was that I've misspelled the
parameter "mapred.tasktracker.map.tasks.maximum".
On Tue, May 24, 2011 at 11:06 AM, Pedro Costa <ps...@gmail.com> wrote:
> I think it's important to say that it exists 2 cpus per node and 12
> core(s) per cpu.
>
> On Tue, May 24, 2011 at 11:02 AM, Pedro Costa <ps...@gmail.com> wrote:
>> And all the nodes have the same configuration. A job has 5000 map tasks.
>>
>> On Tue, May 24, 2011 at 10:57 AM, Pedro Costa <ps...@gmail.com> wrote:
>>> The values are:
>>> #map tasks: 8
>>> #reduce tasks: 10
>>> Map task capacity:10
>>> Reduce task capacity:10
>>>
>>>
>>> On Tue, May 24, 2011 at 8:01 AM, Harsh J <ha...@cloudera.com> wrote:
>>>> How many tasks are present in your job? Do all tasktrackers carry this
>>>> configuration? What is the total reported slot capacity on the JT UI?
>>>>
>>>> On Mon, May 23, 2011 at 10:28 PM, Pedro Costa <ps...@gmail.com> wrote:
>>>>> I think I've to rephrase the question.
>>>>>
>>>>> I set the "mapred.tasktracker.map.tasks.maximum" to 8, hoping that it
>>>>> will run 8*10 map tasks in the whole cluster. But, it only run 8 tasks
>>>>> simultaneously. Why this is happens?
>>>>>
>>>>> On Mon, May 23, 2011 at 5:45 PM, Pedro Costa <ps...@gmail.com> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I'm running hadoop map-reduce in a cluster with 10 machines. I would
>>>>>> like to set in the configuration that each tasktracker can run 8 map
>>>>>> tasks simultaneously and 4 reduce tasks simultaneously. Which
>>>>>> parameters should I configure?
>>>>>>
>>>>>> Thanks,
>>>>>> PSC
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>>
>>>
>>>
>>>
>>
>
>
>
> --
> ---------------------------
> Pedro Sá da Costa
>
> @: pcosta@lasige.di.fc.ul.pt
> @: psdc1978@gmail.com
>
--
---------------------------
Pedro Sá da Costa
@: pcosta@lasige.di.fc.ul.pt
@: psdc1978@gmail.com
Re: How set number of map and reduce can run simultaneously
Posted by Pedro Costa <ps...@gmail.com>.
I think it's important to say that it exists 2 cpus per node and 12
core(s) per cpu.
On Tue, May 24, 2011 at 11:02 AM, Pedro Costa <ps...@gmail.com> wrote:
> And all the nodes have the same configuration. A job has 5000 map tasks.
>
> On Tue, May 24, 2011 at 10:57 AM, Pedro Costa <ps...@gmail.com> wrote:
>> The values are:
>> #map tasks: 8
>> #reduce tasks: 10
>> Map task capacity:10
>> Reduce task capacity:10
>>
>>
>> On Tue, May 24, 2011 at 8:01 AM, Harsh J <ha...@cloudera.com> wrote:
>>> How many tasks are present in your job? Do all tasktrackers carry this
>>> configuration? What is the total reported slot capacity on the JT UI?
>>>
>>> On Mon, May 23, 2011 at 10:28 PM, Pedro Costa <ps...@gmail.com> wrote:
>>>> I think I've to rephrase the question.
>>>>
>>>> I set the "mapred.tasktracker.map.tasks.maximum" to 8, hoping that it
>>>> will run 8*10 map tasks in the whole cluster. But, it only run 8 tasks
>>>> simultaneously. Why this is happens?
>>>>
>>>> On Mon, May 23, 2011 at 5:45 PM, Pedro Costa <ps...@gmail.com> wrote:
>>>>> Hi,
>>>>>
>>>>> I'm running hadoop map-reduce in a cluster with 10 machines. I would
>>>>> like to set in the configuration that each tasktracker can run 8 map
>>>>> tasks simultaneously and 4 reduce tasks simultaneously. Which
>>>>> parameters should I configure?
>>>>>
>>>>> Thanks,
>>>>> PSC
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>>
>
--
---------------------------
Pedro Sá da Costa
@: pcosta@lasige.di.fc.ul.pt
@: psdc1978@gmail.com
Re: How set number of map and reduce can run simultaneously
Posted by Pedro Costa <ps...@gmail.com>.
And all the nodes have the same configuration. A job has 5000 map tasks.
On Tue, May 24, 2011 at 10:57 AM, Pedro Costa <ps...@gmail.com> wrote:
> The values are:
> #map tasks: 8
> #reduce tasks: 10
> Map task capacity:10
> Reduce task capacity:10
>
>
> On Tue, May 24, 2011 at 8:01 AM, Harsh J <ha...@cloudera.com> wrote:
>> How many tasks are present in your job? Do all tasktrackers carry this
>> configuration? What is the total reported slot capacity on the JT UI?
>>
>> On Mon, May 23, 2011 at 10:28 PM, Pedro Costa <ps...@gmail.com> wrote:
>>> I think I've to rephrase the question.
>>>
>>> I set the "mapred.tasktracker.map.tasks.maximum" to 8, hoping that it
>>> will run 8*10 map tasks in the whole cluster. But, it only run 8 tasks
>>> simultaneously. Why this is happens?
>>>
>>> On Mon, May 23, 2011 at 5:45 PM, Pedro Costa <ps...@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> I'm running hadoop map-reduce in a cluster with 10 machines. I would
>>>> like to set in the configuration that each tasktracker can run 8 map
>>>> tasks simultaneously and 4 reduce tasks simultaneously. Which
>>>> parameters should I configure?
>>>>
>>>> Thanks,
>>>> PSC
>>>>
>>>
>>
>>
>>
>> --
>> Harsh J
>>
>
>
>
Re: How set number of map and reduce can run simultaneously
Posted by Pedro Costa <ps...@gmail.com>.
The values are:
#map tasks: 8
#reduce tasks: 10
Map task capacity:10
Reduce task capacity:10
On Tue, May 24, 2011 at 8:01 AM, Harsh J <ha...@cloudera.com> wrote:
> How many tasks are present in your job? Do all tasktrackers carry this
> configuration? What is the total reported slot capacity on the JT UI?
>
> On Mon, May 23, 2011 at 10:28 PM, Pedro Costa <ps...@gmail.com> wrote:
>> I think I've to rephrase the question.
>>
>> I set the "mapred.tasktracker.map.tasks.maximum" to 8, hoping that it
>> will run 8*10 map tasks in the whole cluster. But, it only run 8 tasks
>> simultaneously. Why this is happens?
>>
>> On Mon, May 23, 2011 at 5:45 PM, Pedro Costa <ps...@gmail.com> wrote:
>>> Hi,
>>>
>>> I'm running hadoop map-reduce in a cluster with 10 machines. I would
>>> like to set in the configuration that each tasktracker can run 8 map
>>> tasks simultaneously and 4 reduce tasks simultaneously. Which
>>> parameters should I configure?
>>>
>>> Thanks,
>>> PSC
>>>
>>
>
>
>
> --
> Harsh J
>
--
---------------------------
Pedro Sá da Costa
@: pcosta@lasige.di.fc.ul.pt
@: psdc1978@gmail.com
Re: How set number of map and reduce can run simultaneously
Posted by Harsh J <ha...@cloudera.com>.
How many tasks are present in your job? Do all tasktrackers carry this
configuration? What is the total reported slot capacity on the JT UI?
On Mon, May 23, 2011 at 10:28 PM, Pedro Costa <ps...@gmail.com> wrote:
> I think I've to rephrase the question.
>
> I set the "mapred.tasktracker.map.tasks.maximum" to 8, hoping that it
> will run 8*10 map tasks in the whole cluster. But, it only run 8 tasks
> simultaneously. Why this is happens?
>
> On Mon, May 23, 2011 at 5:45 PM, Pedro Costa <ps...@gmail.com> wrote:
>> Hi,
>>
>> I'm running hadoop map-reduce in a cluster with 10 machines. I would
>> like to set in the configuration that each tasktracker can run 8 map
>> tasks simultaneously and 4 reduce tasks simultaneously. Which
>> parameters should I configure?
>>
>> Thanks,
>> PSC
>>
>
--
Harsh J
Re: How set number of map and reduce can run simultaneously
Posted by Pedro Costa <ps...@gmail.com>.
I think I've to rephrase the question.
I set the "mapred.tasktracker.map.tasks.maximum" to 8, hoping that it
will run 8*10 map tasks in the whole cluster. But, it only run 8 tasks
simultaneously. Why this is happens?
On Mon, May 23, 2011 at 5:45 PM, Pedro Costa <ps...@gmail.com> wrote:
> Hi,
>
> I'm running hadoop map-reduce in a cluster with 10 machines. I would
> like to set in the configuration that each tasktracker can run 8 map
> tasks simultaneously and 4 reduce tasks simultaneously. Which
> parameters should I configure?
>
> Thanks,
> PSC
>