You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ritesh Tijoriwala (JIRA)" <ji...@apache.org> on 2017/06/19 18:30:00 UTC
[jira] [Commented] (SPARK-12414) Remove closure serializer
[ https://issues.apache.org/jira/browse/SPARK-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16054529#comment-16054529 ]
Ritesh Tijoriwala commented on SPARK-12414:
-------------------------------------------
I have a similar situation. I have several classes that I would like to instantiate and use in Executors for e.g. DB connections, Elasticsearch clients, etc. I don't want to write instantiation code in Spark functions and use "statics". There was a nit trick suggested here - https://issues.apache.org/jira/browse/SPARK-650 but seems like this will not work starting 2.0.0 as a consequence of this ticket.
Could anybody from Spark community recommend how to do some initialization on each Spark executor for the job before any task execution begins?
> Remove closure serializer
> -------------------------
>
> Key: SPARK-12414
> URL: https://issues.apache.org/jira/browse/SPARK-12414
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 1.0.0
> Reporter: Andrew Or
> Assignee: Sean Owen
> Fix For: 2.0.0
>
>
> There is a config `spark.closure.serializer` that accepts exactly one value: the java serializer. This is because there are currently bugs in the Kryo serializer that make it not a viable candidate. This was uncovered by an unsuccessful attempt to make it work: SPARK-7708.
> My high level point is that the Java serializer has worked well for at least 6 Spark versions now, and it is an incredibly complicated task to get other serializers (not just Kryo) to work with Spark's closures. IMO the effort is not worth it and we should just remove this documentation and all the code associated with it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org