You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by pr...@nokia.com on 2010/11/23 16:20:10 UTC

OutOfMemory with PFPGrowth

Hello all,
I was able to successfully test PFPGrowth with 50M transactions. Now I am testing with 150M transactions and no matter what group size I use I am getting out of memory when running the FPGrowth job. It finishes parallel counting and transaction sorting job fine but when its running FPGrowth job, I always get outofmemory.

On Hadoop side:map/reduce process heap size is 2G. No. of reduce jobs is 24 on total of 4 hadoop cluster.
On Mahout side: I specified minSupport as 250 and tried with group size from 500 to 3000.
Out of 150M transactions, Its generating about 6500 features so I thought group size of 500 should be good enough to avoid out of memory.

What params can I change to fix the outofmemory issue?
Can someone throw some light on how to come up with optimal parameter values to avoid such issues on production system?

Any help is appreciated.

Praveen

10/11/23 10:16:52 INFO mapred.JobClient:  map 100% reduce 20%
10/11/23 10:17:01 INFO mapred.JobClient:  map 100% reduce 17%
10/11/23 10:17:03 INFO mapred.JobClient: Task Id : attempt_201011221932_0009_r_000013_2, Status : FAILED
Error: Java heap space
10/11/23 10:17:10 INFO mapred.JobClient:  map 100% reduce 14%
10/11/23 10:17:12 INFO mapred.JobClient: Task Id : attempt_201011221932_0009_r_000018_0, Status : FAILED
Error: Java heap space
10/11/23 10:17:14 INFO mapred.JobClient:  map 100% reduce 11%
10/11/23 10:17:16 INFO mapred.JobClient:  map 100% reduce 12%
10/11/23 10:17:16 INFO mapred.JobClient: Task Id : attempt_201011221932_0009_r_000016_1, Status : FAILED
Error: Java heap space
10/11/23 10:17:19 INFO mapred.JobClient:  map 100% reduce 8%
10/11/23 10:17:22 INFO mapred.JobClient:  map 100% reduce 9%
10/11/23 10:17:25 INFO mapred.JobClient: Task Id : attempt_201011221932_0009_r_000019_0, Status : FAILED
Error: Java heap space


RE: OutOfMemory with PFPGrowth

Posted by pr...@nokia.com.
When I set it to 3500 it worked fine. With 3000 I might be falling just in the border. I think I am good for now. Thanks. 

-----Original Message-----
From: ext Robin Anil [mailto:robin.anil@gmail.com] 
Sent: Tuesday, November 23, 2010 11:11 AM
To: user@mahout.apache.org
Subject: Re: OutOfMemory with PFPGrowth

keep it 6500. makes it 1 feature per group. So in which case each reducer gets the conditional tree of the feature. If that can't fit, some pruning may be necessary

Robin

Re: OutOfMemory with PFPGrowth

Posted by Robin Anil <ro...@gmail.com>.
keep it 6500. makes it 1 feature per group. So in which case each reducer
gets the conditional tree of the feature. If that can't fit, some pruning
may be necessary

Robin

RE: OutOfMemory with PFPGrowth

Posted by pr...@nokia.com.
Hi Anil,
Is there a temporary workaround to overcome this issue for now? I am currently stuck with this issue and cannot proceed. In future we have a limit of supporting 0.5 Billion transactions, not just 150M so I really have to come up with a strategy to make it work.

Do you think using 5000 groups would help? I thought after a certain point, increasing the number of groups will not make a difference...

Praveen

-----Original Message-----
From: ext Robin Anil [mailto:robin.anil@gmail.com] 
Sent: Tuesday, November 23, 2010 10:48 AM
To: user@mahout.apache.org
Subject: Re: OutOfMemory with PFPGrowth

The reason is that some groups(having  medium frequency features) have too many branches, when data is very sparse. I have to think if there is a smart pruning mechanism for them. Maybe even compute single conditional tree for one feature for medium sized features.

Robin


On Tue, Nov 23, 2010 at 9:05 PM, <pr...@nokia.com> wrote:

> Yes I tried with 3000 and still the same issue..
>
> -----Original Message-----
> From: ext Robin Anil [mailto:robin.anil@gmail.com]
> Sent: Tuesday, November 23, 2010 10:32 AM
> To: user@mahout.apache.org
> Subject: Re: OutOfMemory with PFPGrowth
>
> Have you tried increasing g to 1000 and above ?
>
> On Tue, Nov 23, 2010 at 8:50 PM, <pr...@nokia.com> wrote:
>
> > Hello all,
> > I was able to successfully test PFPGrowth with 50M transactions. Now 
> > I am testing with 150M transactions and no matter what group size I 
> > use I am getting out of memory when running the FPGrowth job. It 
> > finishes parallel counting and transaction sorting job fine but when 
> > its running FPGrowth job, I always get outofmemory.
> >
> > On Hadoop side:map/reduce process heap size is 2G. No. of reduce 
> > jobs is 24 on total of 4 hadoop cluster.
> > On Mahout side: I specified minSupport as 250 and tried with group 
> > size from 500 to 3000.
> > Out of 150M transactions, Its generating about 6500 features so I 
> > thought group size of 500 should be good enough to avoid out of memory.
> >
> > What params can I change to fix the outofmemory issue?
> > Can someone throw some light on how to come up with optimal 
> > parameter values to avoid such issues on production system?
> >
> > Any help is appreciated.
> >
> > Praveen
> >
> > 10/11/23 10:16:52 INFO mapred.JobClient:  map 100% reduce 20%
> > 10/11/23 10:17:01 INFO mapred.JobClient:  map 100% reduce 17%
> > 10/11/23 10:17:03 INFO mapred.JobClient: Task Id :
> > attempt_201011221932_0009_r_000013_2, Status : FAILED
> > Error: Java heap space
> > 10/11/23 10:17:10 INFO mapred.JobClient:  map 100% reduce 14%
> > 10/11/23 10:17:12 INFO mapred.JobClient: Task Id :
> > attempt_201011221932_0009_r_000018_0, Status : FAILED
> > Error: Java heap space
> > 10/11/23 10:17:14 INFO mapred.JobClient:  map 100% reduce 11%
> > 10/11/23 10:17:16 INFO mapred.JobClient:  map 100% reduce 12%
> > 10/11/23 10:17:16 INFO mapred.JobClient: Task Id :
> > attempt_201011221932_0009_r_000016_1, Status : FAILED
> > Error: Java heap space
> > 10/11/23 10:17:19 INFO mapred.JobClient:  map 100% reduce 8%
> > 10/11/23 10:17:22 INFO mapred.JobClient:  map 100% reduce 9%
> > 10/11/23 10:17:25 INFO mapred.JobClient: Task Id :
> > attempt_201011221932_0009_r_000019_0, Status : FAILED
> > Error: Java heap space
> >
> >
>

Re: OutOfMemory with PFPGrowth

Posted by Robin Anil <ro...@gmail.com>.
The reason is that some groups(having  medium frequency features) have too
many branches, when data is very sparse. I have to think if there is a smart
pruning mechanism for them. Maybe even compute single conditional tree for
one feature for medium sized features.

Robin


On Tue, Nov 23, 2010 at 9:05 PM, <pr...@nokia.com> wrote:

> Yes I tried with 3000 and still the same issue..
>
> -----Original Message-----
> From: ext Robin Anil [mailto:robin.anil@gmail.com]
> Sent: Tuesday, November 23, 2010 10:32 AM
> To: user@mahout.apache.org
> Subject: Re: OutOfMemory with PFPGrowth
>
> Have you tried increasing g to 1000 and above ?
>
> On Tue, Nov 23, 2010 at 8:50 PM, <pr...@nokia.com> wrote:
>
> > Hello all,
> > I was able to successfully test PFPGrowth with 50M transactions. Now I
> > am testing with 150M transactions and no matter what group size I use
> > I am getting out of memory when running the FPGrowth job. It finishes
> > parallel counting and transaction sorting job fine but when its
> > running FPGrowth job, I always get outofmemory.
> >
> > On Hadoop side:map/reduce process heap size is 2G. No. of reduce jobs
> > is 24 on total of 4 hadoop cluster.
> > On Mahout side: I specified minSupport as 250 and tried with group
> > size from 500 to 3000.
> > Out of 150M transactions, Its generating about 6500 features so I
> > thought group size of 500 should be good enough to avoid out of memory.
> >
> > What params can I change to fix the outofmemory issue?
> > Can someone throw some light on how to come up with optimal parameter
> > values to avoid such issues on production system?
> >
> > Any help is appreciated.
> >
> > Praveen
> >
> > 10/11/23 10:16:52 INFO mapred.JobClient:  map 100% reduce 20%
> > 10/11/23 10:17:01 INFO mapred.JobClient:  map 100% reduce 17%
> > 10/11/23 10:17:03 INFO mapred.JobClient: Task Id :
> > attempt_201011221932_0009_r_000013_2, Status : FAILED
> > Error: Java heap space
> > 10/11/23 10:17:10 INFO mapred.JobClient:  map 100% reduce 14%
> > 10/11/23 10:17:12 INFO mapred.JobClient: Task Id :
> > attempt_201011221932_0009_r_000018_0, Status : FAILED
> > Error: Java heap space
> > 10/11/23 10:17:14 INFO mapred.JobClient:  map 100% reduce 11%
> > 10/11/23 10:17:16 INFO mapred.JobClient:  map 100% reduce 12%
> > 10/11/23 10:17:16 INFO mapred.JobClient: Task Id :
> > attempt_201011221932_0009_r_000016_1, Status : FAILED
> > Error: Java heap space
> > 10/11/23 10:17:19 INFO mapred.JobClient:  map 100% reduce 8%
> > 10/11/23 10:17:22 INFO mapred.JobClient:  map 100% reduce 9%
> > 10/11/23 10:17:25 INFO mapred.JobClient: Task Id :
> > attempt_201011221932_0009_r_000019_0, Status : FAILED
> > Error: Java heap space
> >
> >
>

RE: OutOfMemory with PFPGrowth

Posted by pr...@nokia.com.
Yes I tried with 3000 and still the same issue.. 

-----Original Message-----
From: ext Robin Anil [mailto:robin.anil@gmail.com] 
Sent: Tuesday, November 23, 2010 10:32 AM
To: user@mahout.apache.org
Subject: Re: OutOfMemory with PFPGrowth

Have you tried increasing g to 1000 and above ?

On Tue, Nov 23, 2010 at 8:50 PM, <pr...@nokia.com> wrote:

> Hello all,
> I was able to successfully test PFPGrowth with 50M transactions. Now I 
> am testing with 150M transactions and no matter what group size I use 
> I am getting out of memory when running the FPGrowth job. It finishes 
> parallel counting and transaction sorting job fine but when its 
> running FPGrowth job, I always get outofmemory.
>
> On Hadoop side:map/reduce process heap size is 2G. No. of reduce jobs 
> is 24 on total of 4 hadoop cluster.
> On Mahout side: I specified minSupport as 250 and tried with group 
> size from 500 to 3000.
> Out of 150M transactions, Its generating about 6500 features so I 
> thought group size of 500 should be good enough to avoid out of memory.
>
> What params can I change to fix the outofmemory issue?
> Can someone throw some light on how to come up with optimal parameter 
> values to avoid such issues on production system?
>
> Any help is appreciated.
>
> Praveen
>
> 10/11/23 10:16:52 INFO mapred.JobClient:  map 100% reduce 20%
> 10/11/23 10:17:01 INFO mapred.JobClient:  map 100% reduce 17%
> 10/11/23 10:17:03 INFO mapred.JobClient: Task Id :
> attempt_201011221932_0009_r_000013_2, Status : FAILED
> Error: Java heap space
> 10/11/23 10:17:10 INFO mapred.JobClient:  map 100% reduce 14%
> 10/11/23 10:17:12 INFO mapred.JobClient: Task Id :
> attempt_201011221932_0009_r_000018_0, Status : FAILED
> Error: Java heap space
> 10/11/23 10:17:14 INFO mapred.JobClient:  map 100% reduce 11%
> 10/11/23 10:17:16 INFO mapred.JobClient:  map 100% reduce 12%
> 10/11/23 10:17:16 INFO mapred.JobClient: Task Id :
> attempt_201011221932_0009_r_000016_1, Status : FAILED
> Error: Java heap space
> 10/11/23 10:17:19 INFO mapred.JobClient:  map 100% reduce 8%
> 10/11/23 10:17:22 INFO mapred.JobClient:  map 100% reduce 9%
> 10/11/23 10:17:25 INFO mapred.JobClient: Task Id :
> attempt_201011221932_0009_r_000019_0, Status : FAILED
> Error: Java heap space
>
>

Re: OutOfMemory with PFPGrowth

Posted by Robin Anil <ro...@gmail.com>.
Have you tried increasing g to 1000 and above ?

On Tue, Nov 23, 2010 at 8:50 PM, <pr...@nokia.com> wrote:

> Hello all,
> I was able to successfully test PFPGrowth with 50M transactions. Now I am
> testing with 150M transactions and no matter what group size I use I am
> getting out of memory when running the FPGrowth job. It finishes parallel
> counting and transaction sorting job fine but when its running FPGrowth job,
> I always get outofmemory.
>
> On Hadoop side:map/reduce process heap size is 2G. No. of reduce jobs is 24
> on total of 4 hadoop cluster.
> On Mahout side: I specified minSupport as 250 and tried with group size
> from 500 to 3000.
> Out of 150M transactions, Its generating about 6500 features so I thought
> group size of 500 should be good enough to avoid out of memory.
>
> What params can I change to fix the outofmemory issue?
> Can someone throw some light on how to come up with optimal parameter
> values to avoid such issues on production system?
>
> Any help is appreciated.
>
> Praveen
>
> 10/11/23 10:16:52 INFO mapred.JobClient:  map 100% reduce 20%
> 10/11/23 10:17:01 INFO mapred.JobClient:  map 100% reduce 17%
> 10/11/23 10:17:03 INFO mapred.JobClient: Task Id :
> attempt_201011221932_0009_r_000013_2, Status : FAILED
> Error: Java heap space
> 10/11/23 10:17:10 INFO mapred.JobClient:  map 100% reduce 14%
> 10/11/23 10:17:12 INFO mapred.JobClient: Task Id :
> attempt_201011221932_0009_r_000018_0, Status : FAILED
> Error: Java heap space
> 10/11/23 10:17:14 INFO mapred.JobClient:  map 100% reduce 11%
> 10/11/23 10:17:16 INFO mapred.JobClient:  map 100% reduce 12%
> 10/11/23 10:17:16 INFO mapred.JobClient: Task Id :
> attempt_201011221932_0009_r_000016_1, Status : FAILED
> Error: Java heap space
> 10/11/23 10:17:19 INFO mapred.JobClient:  map 100% reduce 8%
> 10/11/23 10:17:22 INFO mapred.JobClient:  map 100% reduce 9%
> 10/11/23 10:17:25 INFO mapred.JobClient: Task Id :
> attempt_201011221932_0009_r_000019_0, Status : FAILED
> Error: Java heap space
>
>