You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-user@hadoop.apache.org by John <jo...@gmail.com> on 2013/11/05 18:17:29 UTC

only one map or reduce job per time on one node

hi,

I have a cluster of 7 nodes. Every node has 2 maps-lots and 1 reduce slot.
Is it possible to force the jobtracker executing only 2 map jobs or 1
reduce job per time? I have found this configuration option:
"mapred.reduce.slowstart.completed.maps". I think this will do exactly what
I want If I set it to 1.0 if there is only one mapreduce job per time. But
whenn there are 2 jobs I think it doesn't work because if the second job is
finished with the map part it will maybe execute a reduce job on node where
still the first job is running. Or am I wrong? Is there a way to allow
executing reduce jobs only if there is no map reduce job on this node?

kind regards

Re: only one map or reduce job per time on one node

Posted by Jens Scheidtmann <je...@gmail.com>.

John,

Why don't you use 1 mapper and 1 reducer?

Am Dienstag, 5. November 2013 schrieb John :

> hi,
>
> I have a cluster of 7 nodes. Every node has 2 maps-lots and 1 reduce slot.
> Is it possible to force the jobtracker executing only 2 map jobs or 1
> reduce job per time?
>

Re: only one map or reduce job per time on one node

Posted by Jens Scheidtmann <je...@gmail.com>.

John,

Why don't you use 1 mapper and 1 reducer?

Am Dienstag, 5. November 2013 schrieb John :

> hi,
>
> I have a cluster of 7 nodes. Every node has 2 maps-lots and 1 reduce slot.
> Is it possible to force the jobtracker executing only 2 map jobs or 1
> reduce job per time?
>

Re: only one map or reduce job per time on one node

Posted by DSuiter RDX <ds...@rdx.com>.

I suspect that the reason no-one is responding with good answers is that
fundamentally, it seems like what you are trying to do runs against the
reason Hadoop is designed the way it is. A parallel process framework is
defeated if you force it to not work concurrently...

Maybe you should look into Oozie, and set a workflow that runs your map
task as a map-only job, and then do the calculation that the reducer would
do in a different program?

There is a Hadoop parameter that allows for task JVM reuse that might solve
your problem also.

Or, add a combiner to your mr job if there is not one already, and then try
dropping the maximum heap sizes of the respective slots?

*Devin Suiter*
Jr. Data Solutions Software Engineer
100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
Google Voice: 412-256-8556 | www.rdx.com

On Wed, Nov 6, 2013 at 5:09 AM, John <jo...@gmail.com> wrote:

> Does no one have an idea if it is possible? In my example the heapsize of
> the tasktracker is 2gb. The max map heap size is 1gb and the mac heap size
> of the reduce process 1,5gb. If there are running 2 maps or just the reduce
> job everything is fine. But if there are running 2 maps + 1 reduce process
> I got a "Error: java.lang.OutOfMemoryError: Java heap space at" message.
>
> kind regards
>
>
> 2013/11/5 John <jo...@gmail.com>
>
>> Because my node swaps the memory if the 2 map slots + 1 reduce is
>> occupied with my job. Sure I can minimize the max memory for the map/reduce
>> process. I tried this already, but I got a out of memory exception if set
>> the max heap size for the map/reduce process to low for my mr job.
>>
>> kind regards
>>
>>
>> 2013/11/5 Vinod Kumar Vavilapalli <vi...@hortonworks.com>
>>
>>> Why do you want to do this?
>>>
>>> +Vinod
>>>
>>> On Nov 5, 2013, at 9:17 AM, John wrote:
>>>
>>> Is it possible to force the jobtracker executing only 2 map jobs or 1
>>> reduce job per time?
>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>

Re: only one map or reduce job per time on one node

Posted by DSuiter RDX <ds...@rdx.com>.

I suspect that the reason no-one is responding with good answers is that
fundamentally, it seems like what you are trying to do runs against the
reason Hadoop is designed the way it is. A parallel process framework is
defeated if you force it to not work concurrently...

Maybe you should look into Oozie, and set a workflow that runs your map
task as a map-only job, and then do the calculation that the reducer would
do in a different program?

There is a Hadoop parameter that allows for task JVM reuse that might solve
your problem also.

Or, add a combiner to your mr job if there is not one already, and then try
dropping the maximum heap sizes of the respective slots?

*Devin Suiter*
Jr. Data Solutions Software Engineer
100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
Google Voice: 412-256-8556 | www.rdx.com

On Wed, Nov 6, 2013 at 5:09 AM, John <jo...@gmail.com> wrote:

> Does no one have an idea if it is possible? In my example the heapsize of
> the tasktracker is 2gb. The max map heap size is 1gb and the mac heap size
> of the reduce process 1,5gb. If there are running 2 maps or just the reduce
> job everything is fine. But if there are running 2 maps + 1 reduce process
> I got a "Error: java.lang.OutOfMemoryError: Java heap space at" message.
>
> kind regards
>
>
> 2013/11/5 John <jo...@gmail.com>
>
>> Because my node swaps the memory if the 2 map slots + 1 reduce is
>> occupied with my job. Sure I can minimize the max memory for the map/reduce
>> process. I tried this already, but I got a out of memory exception if set
>> the max heap size for the map/reduce process to low for my mr job.
>>
>> kind regards
>>
>>
>> 2013/11/5 Vinod Kumar Vavilapalli <vi...@hortonworks.com>
>>
>>> Why do you want to do this?
>>>
>>> +Vinod
>>>
>>> On Nov 5, 2013, at 9:17 AM, John wrote:
>>>
>>> Is it possible to force the jobtracker executing only 2 map jobs or 1
>>> reduce job per time?
>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>

Re: only one map or reduce job per time on one node

Posted by DSuiter RDX <ds...@rdx.com>.

I suspect that the reason no-one is responding with good answers is that
fundamentally, it seems like what you are trying to do runs against the
reason Hadoop is designed the way it is. A parallel process framework is
defeated if you force it to not work concurrently...

Maybe you should look into Oozie, and set a workflow that runs your map
task as a map-only job, and then do the calculation that the reducer would
do in a different program?

There is a Hadoop parameter that allows for task JVM reuse that might solve
your problem also.

Or, add a combiner to your mr job if there is not one already, and then try
dropping the maximum heap sizes of the respective slots?

*Devin Suiter*
Jr. Data Solutions Software Engineer
100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
Google Voice: 412-256-8556 | www.rdx.com

On Wed, Nov 6, 2013 at 5:09 AM, John <jo...@gmail.com> wrote:

> Does no one have an idea if it is possible? In my example the heapsize of
> the tasktracker is 2gb. The max map heap size is 1gb and the mac heap size
> of the reduce process 1,5gb. If there are running 2 maps or just the reduce
> job everything is fine. But if there are running 2 maps + 1 reduce process
> I got a "Error: java.lang.OutOfMemoryError: Java heap space at" message.
>
> kind regards
>
>
> 2013/11/5 John <jo...@gmail.com>
>
>> Because my node swaps the memory if the 2 map slots + 1 reduce is
>> occupied with my job. Sure I can minimize the max memory for the map/reduce
>> process. I tried this already, but I got a out of memory exception if set
>> the max heap size for the map/reduce process to low for my mr job.
>>
>> kind regards
>>
>>
>> 2013/11/5 Vinod Kumar Vavilapalli <vi...@hortonworks.com>
>>
>>> Why do you want to do this?
>>>
>>> +Vinod
>>>
>>> On Nov 5, 2013, at 9:17 AM, John wrote:
>>>
>>> Is it possible to force the jobtracker executing only 2 map jobs or 1
>>> reduce job per time?
>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>