You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@pig.apache.org by shan s <my...@gmail.com> on 2012/04/20 22:18:52 UTC

Java heap space error, for simple load/store on 1.5 gb file

I am simply reading 1.5 gb file with 2458220 records, and storing it back.

I am getting Java Heap space error

The current setting is mapred.child.java.opts -Xmx1073741824



Below is the error from log.

2012-04-20 15:03:15,769 INFO
org.apache.pig.impl.util.SpillableMemoryManager: first memory handler call-
Usage threshold init = 350617600(342400K) used = 522736064(510484K)
committed = 596443136(582464K) max = 715849728(699072K)



Could you please help me interpret this.How do I set space for Java heap.
I am looking at archive on this topic, but didn't see anything similar.
(most of those talk about UDF, but I am simply doing a read/write).

Thanks, Prashant.

Re: Java heap space error, for simple load/store on 1.5 gb file

Posted by Dmitriy Ryaboy <dv...@gmail.com>.

More explicitly: we can't help debug a pig script we haven't seen :).
Send your code, and some sample data if possible.

D

On Fri, Apr 20, 2012 at 8:25 PM, Thejas Nair <th...@hortonworks.com> wrote:
> Which version of pig are you using ?
> Does your query have just a load and store statement ?
> What load and store functions are you using ?
> What would be the largest size of a single tuple in your data ?
>
> Thanks,
> Thejas
>
>
>
> On 4/20/12 1:18 PM, shan s wrote:
>>
>> I am simply reading 1.5 gb file with 2458220 records, and storing it back.
>>
>> I am getting Java Heap space error
>>
>> The current setting is mapred.child.java.opts -Xmx1073741824
>>
>>
>>
>> Below is the error from log.
>>
>> 2012-04-20 15:03:15,769 INFO
>> org.apache.pig.impl.util.SpillableMemoryManager: first memory handler
>> call-
>> Usage threshold init = 350617600(342400K) used = 522736064(510484K)
>> committed = 596443136(582464K) max = 715849728(699072K)
>>
>>
>>
>> Could you please help me interpret this.How do I set space for Java heap.
>> I am looking at archive on this topic, but didn't see anything similar.
>> (most of those talk about UDF, but I am simply doing a read/write).
>>
>> Thanks, Prashant.
>>
>

Re: Java heap space error, for simple load/store on 1.5 gb file

Posted by Thejas Nair <th...@hortonworks.com>.

Which version of pig are you using ?
Does your query have just a load and store statement ?
What load and store functions are you using ?
What would be the largest size of a single tuple in your data ?

Thanks,
Thejas


On 4/20/12 1:18 PM, shan s wrote:
> I am simply reading 1.5 gb file with 2458220 records, and storing it back.
>
> I am getting Java Heap space error
>
> The current setting is mapred.child.java.opts -Xmx1073741824
>
>
>
> Below is the error from log.
>
> 2012-04-20 15:03:15,769 INFO
> org.apache.pig.impl.util.SpillableMemoryManager: first memory handler call-
> Usage threshold init = 350617600(342400K) used = 522736064(510484K)
> committed = 596443136(582464K) max = 715849728(699072K)
>
>
>
> Could you please help me interpret this.How do I set space for Java heap.
> I am looking at archive on this topic, but didn't see anything similar.
> (most of those talk about UDF, but I am simply doing a read/write).
>
> Thanks, Prashant.
>