[Spark Shell] Could the spark shell be reset to the original status?

[Terry Hole <hu...@gmail.com>]

Hi,

Background: The spark shell will get out of memory error after dealing lots
of spark work.

Is there any method which can reset the spark shell to the startup status?
I tried "*:reset*", but it seems not working: i can not create spark
context anymore (some compile error as below) after the "*:reset*". (I have
to restart the shell after OOM to workaround)

== Expanded type of tree ==
TypeRef(TypeSymbol(class $read extends Serializable))
uncaught exception during compilation: java.lang.AssertionError
java.lang.AssertionError: assertion failed: Tried to find '$line16' in
'C:\Users\jhu\AppData\Local\Temp\spark-2ad09490-c0c6-41e2-addb-63087ce0ae63'
but it is not a directory
That entry seems to have slain the compiler.  Shall I replayyour session? I
can re-run each line except the last one.[y/n]
Abandoning crashed session.

Thanks!
-Terry

Re: [Spark Shell] Could the spark shell be reset to the original status?

[Terry Hole <hu...@gmail.com>]

Hi Ted,

Thanks for the information. The post seems little different with my
requirement: suppose we defined different functions to do different
streaming work (e.g. 50 functions), i want to test these 50 functions in
the spark shell, and the shell will always throw OOM at the middle of test
(yes, it could be solved by increasing the jvm memory size, but if we have
more functions, the issue still will happen). The main issue is that the
shell keeps track all the information (class, objects...) from started, so
the java memory will increase time to time when define/invoke the
functions.

Thanks!
- Terry

Ted Yu <yu...@gmail.com>于2015年7月17日周五 下午12:02写道：

> See this recent thread:
>
>
> http://search-hadoop.com/m/q3RTtFW7iMDkrj61/Spark+shell+oom+&subj=java+lang+OutOfMemoryError+PermGen+space
>
>
>
> On Jul 16, 2015, at 8:51 PM, Terry Hole <hu...@gmail.com> wrote:
>
> Hi,
>
> Background: The spark shell will get out of memory error after dealing
> lots of spark work.
>
> Is there any method which can reset the spark shell to the startup status?
> I tried "*:reset*", but it seems not working: i can not create spark
> context anymore (some compile error as below) after the "*:reset*". (I
> have to restart the shell after OOM to workaround)
>
> == Expanded type of tree ==
> TypeRef(TypeSymbol(class $read extends Serializable))
> uncaught exception during compilation: java.lang.AssertionError
> java.lang.AssertionError: assertion failed: Tried to find '$line16' in
> 'C:\Users\jhu\AppData\Local\Temp\spark-2ad09490-c0c6-41e2-addb-63087ce0ae63'
> but it is not a directory
> That entry seems to have slain the compiler.  Shall I replayyour session?
> I can re-run each line except the last one.[y/n]
> Abandoning crashed session.
>
> Thanks!
> -Terry
>
>

Re: [Spark Shell] Could the spark shell be reset to the original status?

[Ted Yu <yu...@gmail.com>]

See this recent thread:

http://search-hadoop.com/m/q3RTtFW7iMDkrj61/Spark+shell+oom+&subj=java+lang+OutOfMemoryError+PermGen+space



> On Jul 16, 2015, at 8:51 PM, Terry Hole <hu...@gmail.com> wrote:
> 
> Hi,
> 
> Background: The spark shell will get out of memory error after dealing lots of spark work. 
> 
> Is there any method which can reset the spark shell to the startup status? I tried ":reset", but it seems not working: i can not create spark context anymore (some compile error as below) after the ":reset". (I have to restart the shell after OOM to workaround) 
> 
> == Expanded type of tree ==
> TypeRef(TypeSymbol(class $read extends Serializable))
> uncaught exception during compilation: java.lang.AssertionError
> java.lang.AssertionError: assertion failed: Tried to find '$line16' in 'C:\Users\jhu\AppData\Local\Temp\spark-2ad09490-c0c6-41e2-addb-63087ce0ae63' but it is not a directory
> That entry seems to have slain the compiler.  Shall I replayyour session? I can re-run each line except the last one.[y/n]
> Abandoning crashed session.
> 
> Thanks!
> -Terry