GC overhead limit exceeded

[Sai Prasanna <an...@gmail.com>]

"java.lang.OutOfMemoryError: GC overhead limit exceeded"

What is the problem. The same code, i run, one instance it runs in 8
second, next time it takes really long time, say 300-500 seconds...
I see the logs a lot of GC overhead limit exceeded is seen. What should be
done ??

Please can someone throw some light on it ??



-- 
*Sai Prasanna. AN*
*II M.Tech (CS), SSSIHL*


*Entire water in the ocean can never sink a ship, Unless it gets inside.All
the pressures of life can never hurt you, Unless you let them in.*

Re: GC overhead limit exceeded

[Nicholas Chammas <ni...@gmail.com>]

But wait, does Spark know to unpersist() RDDs that are not referenced
anywhere? That would’ve taken care of the RDDs that I kept creating and
then orphaning as part of my job testing/profiling.

Is that what SPARK-1103
<https://issues.apache.org/jira/browse/SPARK-1103>is about, btw?

(Sorry to keep digging up this thread.)


On Wed, Apr 16, 2014 at 5:55 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:

> Never mind. I'll take it from both Andrew and Syed's comments that the
> answer is yes. Dunno why I thought otherwise.
>
>
> On Wed, Apr 16, 2014 at 5:43 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> I’m running into a similar issue as the OP. I’m running the same job over
>> and over (with minor tweaks) in the same cluster to profile it. It just
>> recently started throwing java.lang.OutOfMemoryError: Java heap space.
>>
>>> Are you caching a lot of RDD's? If so, maybe you should unpersist() the
>>> ones that you're not using.
>>
>>  I thought that Spark automatically ejects RDDs from the cache using
>> LRU.
>>
>> Do I need to explicitly unpersist() RDDs that are cached with the
>> default storage level?
>>
>> Nick
>>
>>
>> On Thu, Mar 27, 2014 at 1:46 PM, Andrew Or <an...@databricks.com> wrote:
>>
>> Are you caching a lot of RDD's? If so, maybe you should unpersist() the
>>> ones that you're not using. Also, if you're on 0.9, make sure
>>> spark.shuffle.spill is enabled (which it is by default). This allows your
>>> application to spill in-memory content to disk if necessary.
>>>
>>> How much memory are you giving to your executors? The default,
>>> spark.executor.memory is 512m, which is quite low. Consider raising this.
>>> Checking the web UI is a good way to figure out your runtime memory usage.
>>>
>>>
>>> On Thu, Mar 27, 2014 at 9:22 AM, Ognen Duzlevski <
>>> ognen@plainvanillagames.com> wrote:
>>>
>>>>  Look at the tuning guide on Spark's webpage for strategies to cope
>>>> with this.
>>>> I have run into quite a few memory issues like these, some are resolved
>>>> by changing the StorageLevel strategy and employing things like Kryo, some
>>>> are solved by specifying the number of tasks to break down a given
>>>> operation into etc.
>>>>
>>>> Ognen
>>>>
>>>>
>>>> On 3/27/14, 10:21 AM, Sai Prasanna wrote:
>>>>
>>>> "java.lang.OutOfMemoryError: GC overhead limit exceeded"
>>>>
>>>>  What is the problem. The same code, i run, one instance it runs in 8
>>>> second, next time it takes really long time, say 300-500 seconds...
>>>> I see the logs a lot of GC overhead limit exceeded is seen. What should
>>>> be done ??
>>>>
>>>>  Please can someone throw some light on it ??
>>>>
>>>>
>>>>
>>>>  --
>>>>  *Sai Prasanna. AN*
>>>> *II M.Tech (CS), SSSIHL*
>>>>
>>>>
>>>> * Entire water in the ocean can never sink a ship, Unless it gets
>>>> inside. All the pressures of life can never hurt you, Unless you let them
>>>> in.*
>>>>
>>>>
>>>>
>>>
>

Re: GC overhead limit exceeded

[Nicholas Chammas <ni...@gmail.com>]

Never mind. I'll take it from both Andrew and Syed's comments that the
answer is yes. Dunno why I thought otherwise.


On Wed, Apr 16, 2014 at 5:43 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:

> I’m running into a similar issue as the OP. I’m running the same job over
> and over (with minor tweaks) in the same cluster to profile it. It just
> recently started throwing java.lang.OutOfMemoryError: Java heap space.
>
>> Are you caching a lot of RDD's? If so, maybe you should unpersist() the
>> ones that you're not using.
>
>  I thought that Spark automatically ejects RDDs from the cache using LRU.
>
> Do I need to explicitly unpersist() RDDs that are cached with the default
> storage level?
>
> Nick
>
>
> On Thu, Mar 27, 2014 at 1:46 PM, Andrew Or <an...@databricks.com> wrote:
>
> Are you caching a lot of RDD's? If so, maybe you should unpersist() the
>> ones that you're not using. Also, if you're on 0.9, make sure
>> spark.shuffle.spill is enabled (which it is by default). This allows your
>> application to spill in-memory content to disk if necessary.
>>
>> How much memory are you giving to your executors? The default,
>> spark.executor.memory is 512m, which is quite low. Consider raising this.
>> Checking the web UI is a good way to figure out your runtime memory usage.
>>
>>
>> On Thu, Mar 27, 2014 at 9:22 AM, Ognen Duzlevski <
>> ognen@plainvanillagames.com> wrote:
>>
>>>  Look at the tuning guide on Spark's webpage for strategies to cope with
>>> this.
>>> I have run into quite a few memory issues like these, some are resolved
>>> by changing the StorageLevel strategy and employing things like Kryo, some
>>> are solved by specifying the number of tasks to break down a given
>>> operation into etc.
>>>
>>> Ognen
>>>
>>>
>>> On 3/27/14, 10:21 AM, Sai Prasanna wrote:
>>>
>>> "java.lang.OutOfMemoryError: GC overhead limit exceeded"
>>>
>>>  What is the problem. The same code, i run, one instance it runs in 8
>>> second, next time it takes really long time, say 300-500 seconds...
>>> I see the logs a lot of GC overhead limit exceeded is seen. What should
>>> be done ??
>>>
>>>  Please can someone throw some light on it ??
>>>
>>>
>>>
>>>  --
>>>  *Sai Prasanna. AN*
>>> *II M.Tech (CS), SSSIHL*
>>>
>>>
>>> * Entire water in the ocean can never sink a ship, Unless it gets
>>> inside. All the pressures of life can never hurt you, Unless you let them
>>> in.*
>>>
>>>
>>>
>>

Re: GC overhead limit exceeded

[Nicholas Chammas <ni...@gmail.com>]

I’m running into a similar issue as the OP. I’m running the same job over
and over (with minor tweaks) in the same cluster to profile it. It just
recently started throwing java.lang.OutOfMemoryError: Java heap space.

> Are you caching a lot of RDD's? If so, maybe you should unpersist() the
> ones that you're not using.

 I thought that Spark automatically ejects RDDs from the cache using LRU.

Do I need to explicitly unpersist() RDDs that are cached with the default
storage level?

Nick


On Thu, Mar 27, 2014 at 1:46 PM, Andrew Or <an...@databricks.com> wrote:

Are you caching a lot of RDD's? If so, maybe you should unpersist() the
> ones that you're not using. Also, if you're on 0.9, make sure
> spark.shuffle.spill is enabled (which it is by default). This allows your
> application to spill in-memory content to disk if necessary.
>
> How much memory are you giving to your executors? The default,
> spark.executor.memory is 512m, which is quite low. Consider raising this.
> Checking the web UI is a good way to figure out your runtime memory usage.
>
>
> On Thu, Mar 27, 2014 at 9:22 AM, Ognen Duzlevski <
> ognen@plainvanillagames.com> wrote:
>
>>  Look at the tuning guide on Spark's webpage for strategies to cope with
>> this.
>> I have run into quite a few memory issues like these, some are resolved
>> by changing the StorageLevel strategy and employing things like Kryo, some
>> are solved by specifying the number of tasks to break down a given
>> operation into etc.
>>
>> Ognen
>>
>>
>> On 3/27/14, 10:21 AM, Sai Prasanna wrote:
>>
>> "java.lang.OutOfMemoryError: GC overhead limit exceeded"
>>
>>  What is the problem. The same code, i run, one instance it runs in 8
>> second, next time it takes really long time, say 300-500 seconds...
>> I see the logs a lot of GC overhead limit exceeded is seen. What should
>> be done ??
>>
>>  Please can someone throw some light on it ??
>>
>>
>>
>>  --
>>  *Sai Prasanna. AN*
>> *II M.Tech (CS), SSSIHL*
>>
>>
>> * Entire water in the ocean can never sink a ship, Unless it gets inside.
>> All the pressures of life can never hurt you, Unless you let them in.*
>>
>>
>>
>

Re: GC overhead limit exceeded

["Syed A. Hashmi" <sh...@cloudera.com>]

Default is MEMORY_ONLY ... if you explicitly persist a RDD, you have to
explicitly unpersist it if you want to free memory during the job.


On Thu, Mar 27, 2014 at 11:17 PM, Sai Prasanna <an...@gmail.com>wrote:

> Oh sorry, that was a mistake, the default level is MEMORY_ONLY !!
> My doubt was, between two different experiments, are the RDDs cached in
> memory need to be unpersisted???
> Or it doesnt matter ?
>

Re: GC overhead limit exceeded

[Sai Prasanna <an...@gmail.com>]

Oh sorry, that was a mistake, the default level is MEMORY_ONLY !!
My doubt was, between two different experiments, are the RDDs cached in
memory need to be unpersisted???
Or it doesnt matter ?

Re: GC overhead limit exceeded

[Sai Prasanna <an...@gmail.com>]

I dint mention anything, so by default it should be MEMORY_AND_DISK right?

My doubt was, between two different experiments, are the RDDs cached in
memory need to be unpersisted???
Or it doesnt matter ?


On Fri, Mar 28, 2014 at 1:43 AM, Syed A. Hashmi <sh...@cloudera.com>wrote:

> Which storage scheme are you using? I am guessing it is MEMORY_ONLY. In
> large datasets, MEMORY_AND_DISK or MEMORY_AND_DISK_SER work better.
>
> You can call unpersist on an RDD to remove it from Cache though.
>
>
> On Thu, Mar 27, 2014 at 11:57 AM, Sai Prasanna <an...@gmail.com>wrote:
>
>> No i am running on 0.8.1.
>> Yes i am caching a lot, i am benchmarking a simple code in spark where in
>> 512mb, 1g and 2g text files are taken, some basic intermediate operations
>> are done while the intermediate result which will be used in subsequent
>> operations are cached.
>>
>> I thought that, we need not manually unpersist, if i need to cache
>> something and if cache is found full, automatically space will be created
>> by evacuating the earlier. Do i need to unpersist?
>>
>> Moreover, if i run several times, will the previously cached RDDs still
>> remain in the cache? If so can i flush them manually out before the next
>> run? [something like complete cache flush]
>>
>>
>> On Thu, Mar 27, 2014 at 11:16 PM, Andrew Or <an...@databricks.com>wrote:
>>
>>> Are you caching a lot of RDD's? If so, maybe you should unpersist() the
>>> ones that you're not using. Also, if you're on 0.9, make sure
>>> spark.shuffle.spill is enabled (which it is by default). This allows your
>>> application to spill in-memory content to disk if necessary.
>>>
>>> How much memory are you giving to your executors? The default,
>>> spark.executor.memory is 512m, which is quite low. Consider raising this.
>>> Checking the web UI is a good way to figure out your runtime memory usage.
>>>
>>>
>>> On Thu, Mar 27, 2014 at 9:22 AM, Ognen Duzlevski <
>>> ognen@plainvanillagames.com> wrote:
>>>
>>>>  Look at the tuning guide on Spark's webpage for strategies to cope
>>>> with this.
>>>> I have run into quite a few memory issues like these, some are resolved
>>>> by changing the StorageLevel strategy and employing things like Kryo, some
>>>> are solved by specifying the number of tasks to break down a given
>>>> operation into etc.
>>>>
>>>> Ognen
>>>>
>>>>
>>>> On 3/27/14, 10:21 AM, Sai Prasanna wrote:
>>>>
>>>> "java.lang.OutOfMemoryError: GC overhead limit exceeded"
>>>>
>>>>  What is the problem. The same code, i run, one instance it runs in 8
>>>> second, next time it takes really long time, say 300-500 seconds...
>>>> I see the logs a lot of GC overhead limit exceeded is seen. What should
>>>> be done ??
>>>>
>>>>  Please can someone throw some light on it ??
>>>>
>>>>
>>>>
>>>>  --
>>>>  *Sai Prasanna. AN*
>>>> *II M.Tech (CS), SSSIHL*
>>>>
>>>>
>>>> * Entire water in the ocean can never sink a ship, Unless it gets
>>>> inside. All the pressures of life can never hurt you, Unless you let them
>>>> in.*
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> *Sai Prasanna. AN*
>> *II M.Tech (CS), SSSIHL*
>>
>>
>> *Entire water in the ocean can never sink a ship, Unless it gets inside.
>> All the pressures of life can never hurt you, Unless you let them in.*
>>
>
>


-- 
*Sai Prasanna. AN*
*II M.Tech (CS), SSSIHL*


*Entire water in the ocean can never sink a ship, Unless it gets inside.All
the pressures of life can never hurt you, Unless you let them in.*

Re: GC overhead limit exceeded

["Syed A. Hashmi" <sh...@cloudera.com>]

Which storage scheme are you using? I am guessing it is MEMORY_ONLY. In
large datasets, MEMORY_AND_DISK or MEMORY_AND_DISK_SER work better.

You can call unpersist on an RDD to remove it from Cache though.


On Thu, Mar 27, 2014 at 11:57 AM, Sai Prasanna <an...@gmail.com>wrote:

> No i am running on 0.8.1.
> Yes i am caching a lot, i am benchmarking a simple code in spark where in
> 512mb, 1g and 2g text files are taken, some basic intermediate operations
> are done while the intermediate result which will be used in subsequent
> operations are cached.
>
> I thought that, we need not manually unpersist, if i need to cache
> something and if cache is found full, automatically space will be created
> by evacuating the earlier. Do i need to unpersist?
>
> Moreover, if i run several times, will the previously cached RDDs still
> remain in the cache? If so can i flush them manually out before the next
> run? [something like complete cache flush]
>
>
> On Thu, Mar 27, 2014 at 11:16 PM, Andrew Or <an...@databricks.com> wrote:
>
>> Are you caching a lot of RDD's? If so, maybe you should unpersist() the
>> ones that you're not using. Also, if you're on 0.9, make sure
>> spark.shuffle.spill is enabled (which it is by default). This allows your
>> application to spill in-memory content to disk if necessary.
>>
>> How much memory are you giving to your executors? The default,
>> spark.executor.memory is 512m, which is quite low. Consider raising this.
>> Checking the web UI is a good way to figure out your runtime memory usage.
>>
>>
>> On Thu, Mar 27, 2014 at 9:22 AM, Ognen Duzlevski <
>> ognen@plainvanillagames.com> wrote:
>>
>>>  Look at the tuning guide on Spark's webpage for strategies to cope with
>>> this.
>>> I have run into quite a few memory issues like these, some are resolved
>>> by changing the StorageLevel strategy and employing things like Kryo, some
>>> are solved by specifying the number of tasks to break down a given
>>> operation into etc.
>>>
>>> Ognen
>>>
>>>
>>> On 3/27/14, 10:21 AM, Sai Prasanna wrote:
>>>
>>> "java.lang.OutOfMemoryError: GC overhead limit exceeded"
>>>
>>>  What is the problem. The same code, i run, one instance it runs in 8
>>> second, next time it takes really long time, say 300-500 seconds...
>>> I see the logs a lot of GC overhead limit exceeded is seen. What should
>>> be done ??
>>>
>>>  Please can someone throw some light on it ??
>>>
>>>
>>>
>>>  --
>>>  *Sai Prasanna. AN*
>>> *II M.Tech (CS), SSSIHL*
>>>
>>>
>>> * Entire water in the ocean can never sink a ship, Unless it gets
>>> inside. All the pressures of life can never hurt you, Unless you let them
>>> in.*
>>>
>>>
>>>
>>
>
>
> --
> *Sai Prasanna. AN*
> *II M.Tech (CS), SSSIHL*
>
>
> *Entire water in the ocean can never sink a ship, Unless it gets inside.
> All the pressures of life can never hurt you, Unless you let them in.*
>

Re: GC overhead limit exceeded

[Sai Prasanna <an...@gmail.com>]

No i am running on 0.8.1.
Yes i am caching a lot, i am benchmarking a simple code in spark where in
512mb, 1g and 2g text files are taken, some basic intermediate operations
are done while the intermediate result which will be used in subsequent
operations are cached.

I thought that, we need not manually unpersist, if i need to cache
something and if cache is found full, automatically space will be created
by evacuating the earlier. Do i need to unpersist?

Moreover, if i run several times, will the previously cached RDDs still
remain in the cache? If so can i flush them manually out before the next
run? [something like complete cache flush]


On Thu, Mar 27, 2014 at 11:16 PM, Andrew Or <an...@databricks.com> wrote:

> Are you caching a lot of RDD's? If so, maybe you should unpersist() the
> ones that you're not using. Also, if you're on 0.9, make sure
> spark.shuffle.spill is enabled (which it is by default). This allows your
> application to spill in-memory content to disk if necessary.
>
> How much memory are you giving to your executors? The default,
> spark.executor.memory is 512m, which is quite low. Consider raising this.
> Checking the web UI is a good way to figure out your runtime memory usage.
>
>
> On Thu, Mar 27, 2014 at 9:22 AM, Ognen Duzlevski <
> ognen@plainvanillagames.com> wrote:
>
>>  Look at the tuning guide on Spark's webpage for strategies to cope with
>> this.
>> I have run into quite a few memory issues like these, some are resolved
>> by changing the StorageLevel strategy and employing things like Kryo, some
>> are solved by specifying the number of tasks to break down a given
>> operation into etc.
>>
>> Ognen
>>
>>
>> On 3/27/14, 10:21 AM, Sai Prasanna wrote:
>>
>> "java.lang.OutOfMemoryError: GC overhead limit exceeded"
>>
>>  What is the problem. The same code, i run, one instance it runs in 8
>> second, next time it takes really long time, say 300-500 seconds...
>> I see the logs a lot of GC overhead limit exceeded is seen. What should
>> be done ??
>>
>>  Please can someone throw some light on it ??
>>
>>
>>
>>  --
>>  *Sai Prasanna. AN*
>> *II M.Tech (CS), SSSIHL*
>>
>>
>> * Entire water in the ocean can never sink a ship, Unless it gets inside.
>> All the pressures of life can never hurt you, Unless you let them in.*
>>
>>
>>
>


-- 
*Sai Prasanna. AN*
*II M.Tech (CS), SSSIHL*


*Entire water in the ocean can never sink a ship, Unless it gets inside.All
the pressures of life can never hurt you, Unless you let them in.*

Re: GC overhead limit exceeded

[Andrew Or <an...@databricks.com>]

Are you caching a lot of RDD's? If so, maybe you should unpersist() the
ones that you're not using. Also, if you're on 0.9, make sure
spark.shuffle.spill is enabled (which it is by default). This allows your
application to spill in-memory content to disk if necessary.

How much memory are you giving to your executors? The default,
spark.executor.memory is 512m, which is quite low. Consider raising this.
Checking the web UI is a good way to figure out your runtime memory usage.


On Thu, Mar 27, 2014 at 9:22 AM, Ognen Duzlevski <
ognen@plainvanillagames.com> wrote:

>  Look at the tuning guide on Spark's webpage for strategies to cope with
> this.
> I have run into quite a few memory issues like these, some are resolved by
> changing the StorageLevel strategy and employing things like Kryo, some are
> solved by specifying the number of tasks to break down a given operation
> into etc.
>
> Ognen
>
>
> On 3/27/14, 10:21 AM, Sai Prasanna wrote:
>
> "java.lang.OutOfMemoryError: GC overhead limit exceeded"
>
>  What is the problem. The same code, i run, one instance it runs in 8
> second, next time it takes really long time, say 300-500 seconds...
> I see the logs a lot of GC overhead limit exceeded is seen. What should be
> done ??
>
>  Please can someone throw some light on it ??
>
>
>
>  --
>  *Sai Prasanna. AN*
> *II M.Tech (CS), SSSIHL*
>
>
> * Entire water in the ocean can never sink a ship, Unless it gets inside.
> All the pressures of life can never hurt you, Unless you let them in.*
>
>
>

Re: GC overhead limit exceeded

[Ognen Duzlevski <og...@plainvanillagames.com>]

Look at the tuning guide on Spark's webpage for strategies to cope with 
this.
I have run into quite a few memory issues like these, some are resolved 
by changing the StorageLevel strategy and employing things like Kryo, 
some are solved by specifying the number of tasks to break down a given 
operation into etc.

Ognen

On 3/27/14, 10:21 AM, Sai Prasanna wrote:
> "java.lang.OutOfMemoryError: GC overhead limit exceeded"
>
> What is the problem. The same code, i run, one instance it runs in 8 
> second, next time it takes really long time, say 300-500 seconds...
> I see the logs a lot of GC overhead limit exceeded is seen. What 
> should be done ??
>
> Please can someone throw some light on it ??
>
>
>
> -- 
> *Sai Prasanna. AN*
> *II M.Tech (CS), SSSIHL*
> *
> Entire water in the ocean can never sink a ship, Unless it gets inside.
> All the pressures of life can never hurt you, Unless you let them in.*

Re: GC overhead limit exceeded

[Sean Owen <so...@cloudera.com>]

This is another way of Java saying "you ran out of heap space". As
less and less room is available, the GC kicks in more often, freeing
less each time. Before the very last byte of memory is gone, Java may
declare defeat. That's why it's taking so long, and you simply need a
larger heap in whatever is throwing this error.
--
Sean Owen | Director, Data Science | London


On Thu, Mar 27, 2014 at 4:21 PM, Sai Prasanna <an...@gmail.com> wrote:
> "java.lang.OutOfMemoryError: GC overhead limit exceeded"
>
> What is the problem. The same code, i run, one instance it runs in 8 second,
> next time it takes really long time, say 300-500 seconds...
> I see the logs a lot of GC overhead limit exceeded is seen. What should be
> done ??
>
> Please can someone throw some light on it ??
>
>
>
> --
> Sai Prasanna. AN
> II M.Tech (CS), SSSIHL
>
> Entire water in the ocean can never sink a ship, Unless it gets inside.
> All the pressures of life can never hurt you, Unless you let them in.