You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2016/04/26 03:26:12 UTC
[jira] [Commented] (SPARK-14908) Provide support HDFS-located
resources for "spark.executor.extraClasspath" on YARN
[ https://issues.apache.org/jira/browse/SPARK-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15257390#comment-15257390 ]
Apache Spark commented on SPARK-14908:
--------------------------------------
User 'mikhaildubkov' has created a pull request for this issue:
https://github.com/apache/spark/pull/12678
> Provide support HDFS-located resources for "spark.executor.extraClasspath" on YARN
> -----------------------------------------------------------------------------------
>
> Key: SPARK-14908
> URL: https://issues.apache.org/jira/browse/SPARK-14908
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Reporter: Dubkov Mikhail
> Priority: Minor
>
> On our project we use custom implementation of SparkSerializer and we found that it loads serializer class when launch executor (SparkEnv.create()). So, we were forced to use "spark.executor.extraClassPath" and custom serializer class loads fine for now. But, it is not well for deployment process, because currently, "spark.executor.ClassPath" does not support hdfs-based resoruces, that means we should deploy artifact with serializer to each Hadoop node. We would like to simplify deployment process.
> We have tried make changes for this purpose and it works now for us. The changes is relevant only for Hadoop/YARN deployment.
> We didn't any workaround how we can avoid extra class path definition for custom serializer implementation, please, let us know if we missed something.
> I will create pull request for master branch, could you please look into changes and go back with feedback?
> We need these changes in master branch to simplify our future upgrade and I hope this improvement can be helpful for other Spark users.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org