Map JVMs not being terminated after MAP jobs complettion.

[Manu Reddy <ma...@gmail.com>]

Hi,

I am running a memory intensive task on my cluster. The reducer class
should load more than 2gb into memory. I was running into memory issues as
the Map JVMS were not terminated after the completion of map tasks and
still were holding onto memory.

Is this normal? What can I do to ensure that my reduce function gets called
only after all the map jvms are down?


-- 
Manu Reddy,
mobile no: 091-8143405225,
Hyderabad.

Re: Map JVMs not being terminated after MAP jobs complettion.

[Rahul Bhattacharjee <ra...@gmail.com>]

After the map task finishes , the VM terminates and all resources are
reclaimable.

Reduces might start pulling data from completed map tasks much before all
the maps are finished. However there is a property using which you can
delay the reduce step till all the maps are completed.

Rahul


On Fri, Sep 13, 2013 at 12:42 PM, Manu Reddy <ma...@gmail.com> wrote:

> Hi,
>
> I am running a memory intensive task on my cluster. The reducer class
> should load more than 2gb into memory. I was running into memory issues as
> the Map JVMS were not terminated after the completion of map tasks and
> still were holding onto memory.
>
> Is this normal? What can I do to ensure that my reduce function gets
> called only after all the map jvms are down?
>
>
> --
> Manu Reddy,
> mobile no: 091-8143405225,
> Hyderabad.
>

Re: Map JVMs not being terminated after MAP jobs complettion.

[Rahul Bhattacharjee <ra...@gmail.com>]

After the map task finishes , the VM terminates and all resources are
reclaimable.

Reduces might start pulling data from completed map tasks much before all
the maps are finished. However there is a property using which you can
delay the reduce step till all the maps are completed.

Rahul


On Fri, Sep 13, 2013 at 12:42 PM, Manu Reddy <ma...@gmail.com> wrote:

> Hi,
>
> I am running a memory intensive task on my cluster. The reducer class
> should load more than 2gb into memory. I was running into memory issues as
> the Map JVMS were not terminated after the completion of map tasks and
> still were holding onto memory.
>
> Is this normal? What can I do to ensure that my reduce function gets
> called only after all the map jvms are down?
>
>
> --
> Manu Reddy,
> mobile no: 091-8143405225,
> Hyderabad.
>

RE: Map JVMs not being terminated after MAP jobs complettion.

["Kokkula, Sada" <Sa...@bnymellon.com>]

Did you set up the below said parameter during the process?

mapred.job.reduce.memory.mb

From: Manu Reddy [mailto:manubharghav@gmail.com]
Sent: Friday, September 13, 2013 3:12 AM
To: user@hadoop.apache.org
Subject: Map JVMs not being terminated after MAP jobs complettion.

Hi,
I am running a memory intensive task on my cluster. The reducer class should load more than 2gb into memory. I was running into memory issues as the Map JVMS were not terminated after the completion of map tasks and still were holding onto memory.
Is this normal? What can I do to ensure that my reduce function gets called only after all the map jvms are down?


--
Manu Reddy,
mobile no: 091-8143405225,
Hyderabad.

The information contained in this e-mail, and any attachment, is confidential and is intended solely for the use of the intended recipient. Access, copying or re-use of the e-mail or any attachment, or any information contained therein, by any other person is not authorized. If you are not the intended recipient please return the e-mail to the sender and delete it from your computer. Although we attempt to sweep e-mail and attachments for viruses, we do not guarantee that either are virus-free and accept no liability for any damage sustained as a result of viruses. 

Please refer to http://disclaimer.bnymellon.com/eu.htm for certain disclosures relating to European legal entities.

RE: Map JVMs not being terminated after MAP jobs complettion.

["Kokkula, Sada" <Sa...@bnymellon.com>]

Did you set up the below said parameter during the process?

mapred.job.reduce.memory.mb

From: Manu Reddy [mailto:manubharghav@gmail.com]
Sent: Friday, September 13, 2013 3:12 AM
To: user@hadoop.apache.org
Subject: Map JVMs not being terminated after MAP jobs complettion.

Hi,
I am running a memory intensive task on my cluster. The reducer class should load more than 2gb into memory. I was running into memory issues as the Map JVMS were not terminated after the completion of map tasks and still were holding onto memory.
Is this normal? What can I do to ensure that my reduce function gets called only after all the map jvms are down?


--
Manu Reddy,
mobile no: 091-8143405225,
Hyderabad.

The information contained in this e-mail, and any attachment, is confidential and is intended solely for the use of the intended recipient. Access, copying or re-use of the e-mail or any attachment, or any information contained therein, by any other person is not authorized. If you are not the intended recipient please return the e-mail to the sender and delete it from your computer. Although we attempt to sweep e-mail and attachments for viruses, we do not guarantee that either are virus-free and accept no liability for any damage sustained as a result of viruses. 

Please refer to http://disclaimer.bnymellon.com/eu.htm for certain disclosures relating to European legal entities.

Re: Map JVMs not being terminated after MAP jobs complettion.

[Rahul Bhattacharjee <ra...@gmail.com>]

After the map task finishes , the VM terminates and all resources are
reclaimable.

Reduces might start pulling data from completed map tasks much before all
the maps are finished. However there is a property using which you can
delay the reduce step till all the maps are completed.

Rahul


On Fri, Sep 13, 2013 at 12:42 PM, Manu Reddy <ma...@gmail.com> wrote:

> Hi,
>
> I am running a memory intensive task on my cluster. The reducer class
> should load more than 2gb into memory. I was running into memory issues as
> the Map JVMS were not terminated after the completion of map tasks and
> still were holding onto memory.
>
> Is this normal? What can I do to ensure that my reduce function gets
> called only after all the map jvms are down?
>
>
> --
> Manu Reddy,
> mobile no: 091-8143405225,
> Hyderabad.
>

Re: Map JVMs not being terminated after MAP jobs complettion.

[Rahul Bhattacharjee <ra...@gmail.com>]

After the map task finishes , the VM terminates and all resources are
reclaimable.

Reduces might start pulling data from completed map tasks much before all
the maps are finished. However there is a property using which you can
delay the reduce step till all the maps are completed.

Rahul


On Fri, Sep 13, 2013 at 12:42 PM, Manu Reddy <ma...@gmail.com> wrote:

> Hi,
>
> I am running a memory intensive task on my cluster. The reducer class
> should load more than 2gb into memory. I was running into memory issues as
> the Map JVMS were not terminated after the completion of map tasks and
> still were holding onto memory.
>
> Is this normal? What can I do to ensure that my reduce function gets
> called only after all the map jvms are down?
>
>
> --
> Manu Reddy,
> mobile no: 091-8143405225,
> Hyderabad.
>

RE: Map JVMs not being terminated after MAP jobs complettion.

["Kokkula, Sada" <Sa...@bnymellon.com>]

Did you set up the below said parameter during the process?

mapred.job.reduce.memory.mb

From: Manu Reddy [mailto:manubharghav@gmail.com]
Sent: Friday, September 13, 2013 3:12 AM
To: user@hadoop.apache.org
Subject: Map JVMs not being terminated after MAP jobs complettion.

Hi,
I am running a memory intensive task on my cluster. The reducer class should load more than 2gb into memory. I was running into memory issues as the Map JVMS were not terminated after the completion of map tasks and still were holding onto memory.
Is this normal? What can I do to ensure that my reduce function gets called only after all the map jvms are down?


--
Manu Reddy,
mobile no: 091-8143405225,
Hyderabad.

The information contained in this e-mail, and any attachment, is confidential and is intended solely for the use of the intended recipient. Access, copying or re-use of the e-mail or any attachment, or any information contained therein, by any other person is not authorized. If you are not the intended recipient please return the e-mail to the sender and delete it from your computer. Although we attempt to sweep e-mail and attachments for viruses, we do not guarantee that either are virus-free and accept no liability for any damage sustained as a result of viruses. 

Please refer to http://disclaimer.bnymellon.com/eu.htm for certain disclosures relating to European legal entities.

RE: Map JVMs not being terminated after MAP jobs complettion.

["Kokkula, Sada" <Sa...@bnymellon.com>]

Did you set up the below said parameter during the process?

mapred.job.reduce.memory.mb

From: Manu Reddy [mailto:manubharghav@gmail.com]
Sent: Friday, September 13, 2013 3:12 AM
To: user@hadoop.apache.org
Subject: Map JVMs not being terminated after MAP jobs complettion.

Hi,
I am running a memory intensive task on my cluster. The reducer class should load more than 2gb into memory. I was running into memory issues as the Map JVMS were not terminated after the completion of map tasks and still were holding onto memory.
Is this normal? What can I do to ensure that my reduce function gets called only after all the map jvms are down?


--
Manu Reddy,
mobile no: 091-8143405225,
Hyderabad.

The information contained in this e-mail, and any attachment, is confidential and is intended solely for the use of the intended recipient. Access, copying or re-use of the e-mail or any attachment, or any information contained therein, by any other person is not authorized. If you are not the intended recipient please return the e-mail to the sender and delete it from your computer. Although we attempt to sweep e-mail and attachments for viruses, we do not guarantee that either are virus-free and accept no liability for any damage sustained as a result of viruses. 

Please refer to http://disclaimer.bnymellon.com/eu.htm for certain disclosures relating to European legal entities.