You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2018/10/26 19:12:00 UTC
[jira] [Commented] (SPARK-25839) Implement use of KryoPool in
KryoSerializer
[ https://issues.apache.org/jira/browse/SPARK-25839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665574#comment-16665574 ]
Apache Spark commented on SPARK-25839:
--------------------------------------
User 'patrickbrownsync' has created a pull request for this issue:
https://github.com/apache/spark/pull/22855
> Implement use of KryoPool in KryoSerializer
> -------------------------------------------
>
> Key: SPARK-25839
> URL: https://issues.apache.org/jira/browse/SPARK-25839
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.0.2, 2.3.1, 2.3.2
> Reporter: Patrick Brown
> Priority: Minor
>
> The current implementation of KryoSerializer does not use KryoPool, which is recommended by Kryo due to the creation of a Kryo instance being slow.
>
> The current implementation references the KryoSerializerInstance private variable cachedKryo as effectively being a pool of size 1. However (in my admittedly somewhat limited research) it seems that frequently (such as in the ClosureCleaner ensureSerializable method) a new instance of KryoSerializerInstance is created, which in turn forces a new instance of Kryo itself to be created, this instance is then dropped from scope, causing the "pool" not to be re-used.
>
> I have a small set of proposed changes we have been using on an internal production application (running 24x7 for 6+ months, processing 10k+ jobs a day) which implements using a KryoPool inside KryoSerializer which is then used by each KryoSerializerInstance to borrow a Kryo instance.
>
> I believe this is mainly a performance improvement for applications processing a large number of small jobs, where the cost of instantiating Kryo instances is a larger portion of execution time compared to larger jobs.
>
> I have discussed this proposed change in the dev mailing list and it was suggested I create this issue and a PR. It was also suggested I accompany that with some performance metrics, which it is my plan to do.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org