[jira] [Commented] (TOREE-430) ClassNotFoundException with lambdas on CDH 5.10.0

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TOREE-430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119632#comment-16119632 ] 

ASF GitHub Bot commented on TOREE-430:
--------------------------------------

GitHub user Myllyenko opened a pull request:

    https://github.com/apache/incubator-toree/pull/129

    Fix for TOREE-430

    To fix TOREE-430, REPL-generated classes must be made visible to a classloader.
    
    The only working way I found is to set the `spark.repl.class.outputDir` system property (and subsequently the corresponding Spark configuration property) to a path to an output directory of a REPL compiler. To achieve that, I had to move `SparkIMain` initialisation before creation of `SparkContext`.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/Myllyenko/incubator-toree TOREE-430

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-toree/pull/129.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #129
    
----
commit 2de78a0c50237bd2166843fd89982022d45a41b0
Author: Igor Melnichenko <my...@ya.ru>
Date:   2017-08-04T11:11:13Z

    Fixed TOREE-430 by setting the spark.repl.class.outputDir system property to a path to an output directory of a REPL compiler in the Scala 2.10 environment.
    
    (cherry picked from commit 72261a6eb9759b0f0cae89793fdac25976d10c48)
    
    # Conflicts:
    #	scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala

----


> ClassNotFoundException with lambdas on CDH 5.10.0
> -------------------------------------------------
>
>                 Key: TOREE-430
>                 URL: https://issues.apache.org/jira/browse/TOREE-430
>             Project: TOREE
>          Issue Type: Bug
>          Components: Kernel
>    Affects Versions: 0.1.0
>         Environment: CDH 5.10.0 (Spark 1.6.0, Scala 2.10.5)
>            Reporter: Igor Melnichenko
>
> In the specified environment, Toree classloader fails to load classes defined and compiled in REPL. For instance, the following Spark example fails with ClassNotFoundException and works fine if the lambda is removed:
> {code:title=Example.scala|borderStyle=solid}
> val NUM_SAMPLES = 100
> val count = sc.parallelize(1 to NUM_SAMPLES).filter { _ =>
>   val x = math.random
>   val y = math.random
>   x*x + y*y < 1
> }.count()
> println(s"Pi is roughly ${4.0 * count / NUM_SAMPLES}")
> {code}
> Cause: a location of a directory used by a REPL compiler to store runtime-compiled classes is different from a location where classloader tries to find these classes.
> PR with fix will be opened soon.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)