[jira] [Updated] (SPARK-18646) ExecutorClassLoader for spark-shell does not honor spark.executor.userClassPathFirst

["Min Shen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-18646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Min Shen updated SPARK-18646:
-----------------------------
    Component/s:     (was: Spark Core)
                 Spark Shell

> ExecutorClassLoader for spark-shell does not honor spark.executor.userClassPathFirst
> ------------------------------------------------------------------------------------
>
>                 Key: SPARK-18646
>                 URL: https://issues.apache.org/jira/browse/SPARK-18646
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Shell
>    Affects Versions: 1.6.2
>            Reporter: Min Shen
>
> When submitting a spark-shell application, the executor side classloader is set to be {{ExecutorClassLoader}}.
> However, it appears that when {{ExecutorClassLoader}} is used, parameter {{spark.executor.userClassPathFirst}} is not honored.
> It turns out that, since {{ExecutorClassLoader}} class is defined as
> {noformat}
> class ExecutorClassLoader(conf: SparkConf, classUri: String, parent: ClassLoader,
>     userClassPathFirst: Boolean) extends ClassLoader with Logging
> {noformat}
> its parent classloader is actually the system default classloader (due to {{ClassLoader}} class's default constructor) rather than the "parent" classloader specified in {{ExecutorClassLoader}}'s constructor.
> As a result, when {{spark.executor.userClassPathFirst}} is set to true, even though the "parent" classloader is {{ChildFirstURLClassLoader}}, {{ExecutorClassLoader.getParent()}} will return the system default classloader.
> Thus, when {{ExecutorClassLoader}} tries to load a class, it will first attempt to load it through the system default classloader, and this will break the {{spark.executor.userClassPathFirst}} behavior.
> A simple fix would be to define {{ExecutorClassLoader}} as:
> {noformat}
> class ExecutorClassLoader(conf: SparkConf, classUri: String, parent: ClassLoader,
>     userClassPathFirst: Boolean) extends ClassLoader(parent) with Logging
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org