You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by xa...@orange-ftgroup.com on 2009/08/05 20:44:40 UTC
OutofMemoryError
Hi Everybody,
I have this error PIG-766
(https://issues.apache.org/jira/browse/PIG-766). I wonder if somebody
fix this issue or is there some recommendations related to this issue.
Xavier
RE: OutofMemoryError
Posted by xa...@orange-ftgroup.com.
Thank you, it works.
X
-----Original Message-----
From: Mridul Muralidharan [mailto:mridulm@yahoo-inc.com]
Sent: Wednesday, August 05, 2009 2:21 PM
To: pig-user@hadoop.apache.org
Subject: Re: OutofMemoryError
You might get better results if you try :
X=load '$path' using PigSotrage(',') as (f1,(around 30 fields))
Name = group X by f1;
name1 = foreach Name generate group,COUNT($1);
namekey = filter name1 by group != '';
store namekey into '$STOREDIR' using PigStorage();
Essentially : group followed by an algebraic foreach is combinable - and
so the output per reducer is drastically lowered.
Regards,
Mridul
xavier.quintuna@orange-ftgroup.com wrote:
> Hi Olga,
>
> My script is simple:
>
> X=load '$path' using PigSotrage(',') as (f1,(around 30 fields))
>
> Name= group X by f1;
>
> nameF = filter name by group.f1 != '';
>
> namekey = foreach nameF generate FLATTEN($0),COUNT($1);
>
> store namekey into '$STOREDIR' using PigStorage();
>
> I'm going to check the PigUserCookBook. I have around 5000 tar files.
>
> Please feel free to point some errors or sugestions in my script.
>
> Xavier
>
>
> -----Original Message-----
> From: Olga Natkovich [mailto:olgan@yahoo-inc.com]
> Sent: Wednesday, August 05, 2009 12:39 PM
> To: pig-user@hadoop.apache.org
> Subject: RE: OutofMemoryError
>
> Out of memory condition can be caused by a number of different issue.
> You might want to look at http://wiki.apache.org/pig/PigUserCookbook
to
> see if any performance enhancements resolve your issue.
>
> It is also a good idea to post your script to see if others could
> provide help.
>
> Olga
>
> -----Original Message-----
> From: xavier.quintuna@orange-ftgroup.com
> [mailto:xavier.quintuna@orange-ftgroup.com]
> Sent: Wednesday, August 05, 2009 11:45 AM
> To: pig-user@hadoop.apache.org
> Subject: OutofMemoryError
>
> Hi Everybody,
>
> I have this error PIG-766
> (https://issues.apache.org/jira/browse/PIG-766). I wonder if somebody
> fix this issue or is there some recommendations related to this issue.
>
> Xavier
Re: OutofMemoryError
Posted by Mridul Muralidharan <mr...@yahoo-inc.com>.
You might get better results if you try :
X=load '$path' using PigSotrage(',') as (f1,(around 30 fields))
Name = group X by f1;
name1 = foreach Name generate group,COUNT($1);
namekey = filter name1 by group != '';
store namekey into '$STOREDIR' using PigStorage();
Essentially : group followed by an algebraic foreach is combinable - and
so the output per reducer is drastically lowered.
Regards,
Mridul
xavier.quintuna@orange-ftgroup.com wrote:
> Hi Olga,
>
> My script is simple:
>
> X=load '$path' using PigSotrage(',') as (f1,(around 30 fields))
>
> Name= group X by f1;
>
> nameF = filter name by group.f1 != '';
>
> namekey = foreach nameF generate FLATTEN($0),COUNT($1);
>
> store namekey into '$STOREDIR' using PigStorage();
>
> I'm going to check the PigUserCookBook. I have around 5000 tar files.
>
> Please feel free to point some errors or sugestions in my script.
>
> Xavier
>
>
> -----Original Message-----
> From: Olga Natkovich [mailto:olgan@yahoo-inc.com]
> Sent: Wednesday, August 05, 2009 12:39 PM
> To: pig-user@hadoop.apache.org
> Subject: RE: OutofMemoryError
>
> Out of memory condition can be caused by a number of different issue.
> You might want to look at http://wiki.apache.org/pig/PigUserCookbook to
> see if any performance enhancements resolve your issue.
>
> It is also a good idea to post your script to see if others could
> provide help.
>
> Olga
>
> -----Original Message-----
> From: xavier.quintuna@orange-ftgroup.com
> [mailto:xavier.quintuna@orange-ftgroup.com]
> Sent: Wednesday, August 05, 2009 11:45 AM
> To: pig-user@hadoop.apache.org
> Subject: OutofMemoryError
>
> Hi Everybody,
>
> I have this error PIG-766
> (https://issues.apache.org/jira/browse/PIG-766). I wonder if somebody
> fix this issue or is there some recommendations related to this issue.
>
> Xavier
RE: OutofMemoryError
Posted by xa...@orange-ftgroup.com.
Hi Olga,
My script is simple:
X=load '$path' using PigSotrage(',') as (f1,(around 30 fields))
Name= group X by f1;
nameF = filter name by group.f1 != '';
namekey = foreach nameF generate FLATTEN($0),COUNT($1);
store namekey into '$STOREDIR' using PigStorage();
I'm going to check the PigUserCookBook. I have around 5000 tar files.
Please feel free to point some errors or sugestions in my script.
Xavier
-----Original Message-----
From: Olga Natkovich [mailto:olgan@yahoo-inc.com]
Sent: Wednesday, August 05, 2009 12:39 PM
To: pig-user@hadoop.apache.org
Subject: RE: OutofMemoryError
Out of memory condition can be caused by a number of different issue.
You might want to look at http://wiki.apache.org/pig/PigUserCookbook to
see if any performance enhancements resolve your issue.
It is also a good idea to post your script to see if others could
provide help.
Olga
-----Original Message-----
From: xavier.quintuna@orange-ftgroup.com
[mailto:xavier.quintuna@orange-ftgroup.com]
Sent: Wednesday, August 05, 2009 11:45 AM
To: pig-user@hadoop.apache.org
Subject: OutofMemoryError
Hi Everybody,
I have this error PIG-766
(https://issues.apache.org/jira/browse/PIG-766). I wonder if somebody
fix this issue or is there some recommendations related to this issue.
Xavier
RE: OutofMemoryError
Posted by Olga Natkovich <ol...@yahoo-inc.com>.
Out of memory condition can be caused by a number of different issue.
You might want to look at http://wiki.apache.org/pig/PigUserCookbook to
see if any performance enhancements resolve your issue.
It is also a good idea to post your script to see if others could
provide help.
Olga
-----Original Message-----
From: xavier.quintuna@orange-ftgroup.com
[mailto:xavier.quintuna@orange-ftgroup.com]
Sent: Wednesday, August 05, 2009 11:45 AM
To: pig-user@hadoop.apache.org
Subject: OutofMemoryError
Hi Everybody,
I have this error PIG-766
(https://issues.apache.org/jira/browse/PIG-766). I wonder if somebody
fix this issue or is there some recommendations related to this issue.
Xavier