You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Chandraprakash Bhagtani <cp...@gmail.com> on 2009/11/30 13:14:06 UTC
Problem with mapred.job.reuse.jvm.num.tasks
Hi,
In one of my hadoop jobs I had set mapred.job.reuse.jvm.num.tasks = -1. The
job was such that each map/reduce
task occupied around 2GB RAM. So by setting mapred.job.reuse.jvm.num.tasks =
-1, even when all the maps had
finished, JVMs were still alive. Though the JVMs were not using CPU, but
occupied RAM. Thus reducing overall available
RAM for reducer JVMs. All the JVMs were killed when the Job completed.
Is this a bug in hadoop? If yes, should I open a JIRA for that?
Please enlighten, Thanks in advance.
--
Thanks & Regards,
Chandra Prakash Bhagtani
Re: 0.20 ConcurrentModificationException
Posted by Todd Lipcon <to...@cloudera.com>.
On Wed, Dec 2, 2009 at 8:46 AM, Arv Mistry <ar...@kindsight.net> wrote:
> Thanks Todd, I'm not sure what you mean ' Cloudera's distribution?'
> Is that a separate build for hadoop? If so please send me the link and I
> will try it
>
>
Yes - like Redhat or Ubuntu provides distributions of Linux, we provide a
distro of Hadoop. You can get it from http://archive.cloudera.com/. If
you're used to the Apache tarballs, the CDH tarball should be a drop-in
replacement.
If you'd prefer to stick with Apache, you can manually apply the patch from
that JIRA and rebuild Hadoop.
If it turns out that the problem sticks around, please report back or file a
JIRA.
Thanks
-Todd
> Cheers Arv
>
> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: December 2, 2009 11:39 AM
> To: common-user@hadoop.apache.org
> Subject: Re: 0.20 ConcurrentModificationException
>
> Certainly looks like HADOOP-6269 to me.
>
> Can you try Cloudera's distribution?This patch is included.
>
> -Todd
>
> On Wed, Dec 2, 2009 at 4:23 AM, Arv Mistry <ar...@kindsight.net> wrote:
>
> > Hi,
> >
> > I've recently upgraded hadoop to 0.20 and am seeing this concurrent
> mod
> > exception on startup which I never got in 0.19.
> > Is this a known bug in 0.20? I did see this JIRA report, not sure its
> > related http://issues.apache.org/jira/browse/HADOOP-6269
> >
> > Is there a workaround or should I be getting the FS a different way in
> > 0.20?
> >
> > java.util.ConcurrentModificationException
> > at java.util.AbstractList$Itr.checkForComodification(Unknown
> Source)
> > at java.util.AbstractList$Itr.next(Unknown Source)
> > at
> >
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:10
> > 28)
> > at
> > org.apache.hadoop.conf.Configuration.getProps(Configuration.java:979)
> > at org.apache.hadoop.conf.Configuration.get(Configuration.java:435)
> > at
> > org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:103)
> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
> > at
> > com.rialto.hadoop.HadoopFileWriter.<init>(HadoopFileWriter.java:66)
> >
> > Cheers Arv
> >
> >
> >
>
RE: 0.20 ConcurrentModificationException
Posted by Arv Mistry <ar...@kindsight.net>.
Thanks Todd, I'm not sure what you mean ' Cloudera's distribution?'
Is that a separate build for hadoop? If so please send me the link and I
will try it
Cheers Arv
-----Original Message-----
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: December 2, 2009 11:39 AM
To: common-user@hadoop.apache.org
Subject: Re: 0.20 ConcurrentModificationException
Certainly looks like HADOOP-6269 to me.
Can you try Cloudera's distribution?This patch is included.
-Todd
On Wed, Dec 2, 2009 at 4:23 AM, Arv Mistry <ar...@kindsight.net> wrote:
> Hi,
>
> I've recently upgraded hadoop to 0.20 and am seeing this concurrent
mod
> exception on startup which I never got in 0.19.
> Is this a known bug in 0.20? I did see this JIRA report, not sure its
> related http://issues.apache.org/jira/browse/HADOOP-6269
>
> Is there a workaround or should I be getting the FS a different way in
> 0.20?
>
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(Unknown
Source)
> at java.util.AbstractList$Itr.next(Unknown Source)
> at
>
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:10
> 28)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:979)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:435)
> at
> org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:103)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
> at
> com.rialto.hadoop.HadoopFileWriter.<init>(HadoopFileWriter.java:66)
>
> Cheers Arv
>
>
>
Re: 0.20 ConcurrentModificationException
Posted by Todd Lipcon <to...@cloudera.com>.
Certainly looks like HADOOP-6269 to me.
Can you try Cloudera's distribution?This patch is included.
-Todd
On Wed, Dec 2, 2009 at 4:23 AM, Arv Mistry <ar...@kindsight.net> wrote:
> Hi,
>
> I've recently upgraded hadoop to 0.20 and am seeing this concurrent mod
> exception on startup which I never got in 0.19.
> Is this a known bug in 0.20? I did see this JIRA report, not sure its
> related http://issues.apache.org/jira/browse/HADOOP-6269
>
> Is there a workaround or should I be getting the FS a different way in
> 0.20?
>
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(Unknown Source)
> at java.util.AbstractList$Itr.next(Unknown Source)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:10
> 28)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:979)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:435)
> at
> org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:103)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
> at
> com.rialto.hadoop.HadoopFileWriter.<init>(HadoopFileWriter.java:66)
>
> Cheers Arv
>
>
>
0.20 ConcurrentModificationException
Posted by Arv Mistry <ar...@kindsight.net>.
Hi,
I've recently upgraded hadoop to 0.20 and am seeing this concurrent mod
exception on startup which I never got in 0.19.
Is this a known bug in 0.20? I did see this JIRA report, not sure its
related http://issues.apache.org/jira/browse/HADOOP-6269
Is there a workaround or should I be getting the FS a different way in
0.20?
java.util.ConcurrentModificationException
at java.util.AbstractList$Itr.checkForComodification(Unknown Source)
at java.util.AbstractList$Itr.next(Unknown Source)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:10
28)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:979)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:435)
at
org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:103)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at
com.rialto.hadoop.HadoopFileWriter.<init>(HadoopFileWriter.java:66)
Cheers Arv
Re: Problem with mapred.job.reuse.jvm.num.tasks
Posted by Chandraprakash Bhagtani <cp...@gmail.com>.
Thanks for the reply Amogh.
Yes, my issue is similar to the pointer you have given. I just want to say
that as soon as the map tasks finish,
all map task JVMs should be killed since they are no more required.
On Mon, Nov 30, 2009 at 6:40 PM, Amogh Vasekar <am...@yahoo-inc.com> wrote:
> Hi,
> Task slots reuse JVM over the course of entire job right? Specifically,
> would like to point to :
>
> http://issues.apache.org/jira/browse/MAPREDUCE-453?focusedCommentId=12619492&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12619492
>
> Thanks,
> Amogh
>
> On 11/30/09 5:44 PM, "Chandraprakash Bhagtani" <cp...@gmail.com>
> wrote:
>
> Hi,
>
> In one of my hadoop jobs I had set mapred.job.reuse.jvm.num.tasks = -1. The
> job was such that each map/reduce
> task occupied around 2GB RAM. So by setting mapred.job.reuse.jvm.num.tasks
> =
> -1, even when all the maps had
> finished, JVMs were still alive. Though the JVMs were not using CPU, but
> occupied RAM. Thus reducing overall available
> RAM for reducer JVMs. All the JVMs were killed when the Job completed.
>
> Is this a bug in hadoop? If yes, should I open a JIRA for that?
>
> Please enlighten, Thanks in advance.
>
> --
> Thanks & Regards,
> Chandra Prakash Bhagtani
>
>
--
Thanks & Regards,
Chandra Prakash Bhagtani,
Re: Problem with mapred.job.reuse.jvm.num.tasks
Posted by Amogh Vasekar <am...@yahoo-inc.com>.
Hi,
Task slots reuse JVM over the course of entire job right? Specifically, would like to point to :
http://issues.apache.org/jira/browse/MAPREDUCE-453?focusedCommentId=12619492&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12619492
Thanks,
Amogh
On 11/30/09 5:44 PM, "Chandraprakash Bhagtani" <cp...@gmail.com> wrote:
Hi,
In one of my hadoop jobs I had set mapred.job.reuse.jvm.num.tasks = -1. The
job was such that each map/reduce
task occupied around 2GB RAM. So by setting mapred.job.reuse.jvm.num.tasks =
-1, even when all the maps had
finished, JVMs were still alive. Though the JVMs were not using CPU, but
occupied RAM. Thus reducing overall available
RAM for reducer JVMs. All the JVMs were killed when the Job completed.
Is this a bug in hadoop? If yes, should I open a JIRA for that?
Please enlighten, Thanks in advance.
--
Thanks & Regards,
Chandra Prakash Bhagtani