[jira] [Created] (SPARK-18646) ExecutorClassLoader for spark-shell does not honor spark.executor.userClassPathFirst

["Min Shen (JIRA)" <ji...@apache.org>]

Min Shen created SPARK-18646:
--------------------------------

             Summary: ExecutorClassLoader for spark-shell does not honor spark.executor.userClassPathFirst
                 Key: SPARK-18646
                 URL: https://issues.apache.org/jira/browse/SPARK-18646
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 1.6.2
            Reporter: Min Shen


When submitting a spark-shell application, the executor side classloader is set to be {{ExecutorClassLoader}}.
However, it appears that when {{ExecutorClassLoader}} is used, parameter {{spark.executor.userClassPathFirst}} is not honored.
It turns out that, since {{ExecutorClassLoader}} class is defined as
{noformat}
class ExecutorClassLoader(conf: SparkConf, classUri: String, parent: ClassLoader,
    userClassPathFirst: Boolean) extends ClassLoader with Logging
{noformat}
its parent classloader is actually the system default classloader (due to {{ClassLoader}} class's default constructor) rather than the "parent" classloader specified in {{ExecutorClassLoader}}'s constructor.
As a result, when {{spark.executor.userClassPathFirst}} is set to true, even though the "parent" classloader is {{ChildFirstURLClassLoader}}, {{ExecutorClassLoader.getParent()}} will return the system default classloader.
Thus, when {{ExecutorClassLoader}} tries to load a class, it will first attempt to load it through the system default classloader, and this will break the {{spark.executor.userClassPathFirst}} behavior.

A simple fix would be to define {{ExecutorClassLoader}} as:
{noformat}
class ExecutorClassLoader(conf: SparkConf, classUri: String, parent: ClassLoader,
    userClassPathFirst: Boolean) extends ClassLoader(parent) with Logging
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org