[jira] [Created] (SPARK-3166) Custom serialisers can't be shipped in application jars

["Graham Dennis (JIRA)" <ji...@apache.org>]

Graham Dennis created SPARK-3166:
------------------------------------

             Summary: Custom serialisers can't be shipped in application jars
                 Key: SPARK-3166
                 URL: https://issues.apache.org/jira/browse/SPARK-3166
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 1.0.2
            Reporter: Graham Dennis


Spark cannot currently use a custom serialiser that is shipped with the application jar. Trying to do this causes a java.lang.ClassNotFoundException when trying to instantiate the custom serialiser in the Executor processes. This occurs because Spark attempts to instantiate the custom serialiser before the application jar has been shipped to the Executor process. A reproduction of the problem is available here: https://github.com/GrahamDennis/spark-custom-serialiser

I've verified this problem in Spark 1.0.2, and Spark master and 1.1 branches as of August 21, 2014.  This issue is related to SPARK-2878, and my fix for that issue (https://github.com/apache/spark/pull/1890) also solves this.  My pull request was not merged because it adds the user jar to the Executor processes' class path at launch time.  Such a significant change was thought by [~rxin] to require more QA, and should be considered for inclusion in 1.2 at the earliest.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org