You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Patrick Brown (JIRA)" <ji...@apache.org> on 2018/10/25 19:24:00 UTC

[jira] [Created] (SPARK-25839) Implement use of KryoPool in KryoSerializer

Patrick Brown created SPARK-25839:
-------------------------------------

             Summary: Implement use of KryoPool in KryoSerializer
                 Key: SPARK-25839
                 URL: https://issues.apache.org/jira/browse/SPARK-25839
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 2.3.2, 2.3.1, 2.0.2
            Reporter: Patrick Brown


The current implementation of KryoSerializer does not use KryoPool, which is recommended by Kryo due to the creation of a Kryo instance being slow.

 

The current implementation references the KryoSerializerInstance private variable cachedKryo as effectively being a pool of size 1. However (in my admittedly somewhat limited research) it seems that frequently (such as in the ClosureCleaner ensureSerializable method) a new instance of KryoSerializerInstance is created, which in turn forces a new instance of Kryo itself to be created, this instance is then dropped from scope, causing the "pool" not to be re-used.

 

I have a small set of proposed changes we have been using on an internal production application (running 24x7 for 6+ months, processing 10k+ jobs a day) which implements using a KryoPool inside KryoSerializer which is then used by each KryoSerializerInstance to borrow a Kryo instance.

 

I believe this is mainly a performance improvement for applications processing a large number of small jobs, where the cost of instantiating Kryo instances is a larger portion of execution time compared to larger jobs.

 

I have discussed this proposed change in the dev mailing list and it was suggested I create this issue and a PR. It was also suggested I accompany that with some performance metrics, which it is my plan to do.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org