[jira] [Resolved] (SPARK-12473) Reuse serializer instances for performance

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-12473.
----------------------------------
    Resolution: Incomplete

> Reuse serializer instances for performance
> ------------------------------------------
>
>                 Key: SPARK-12473
>                 URL: https://issues.apache.org/jira/browse/SPARK-12473
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 1.6.0
>            Reporter: Andrew Or
>            Assignee: Andrew Or
>            Priority: Minor
>              Labels: bulk-closed
>
> After commit de02782 of page rank regressed from 242s to 260s, about 7%. Although currently it's only 7%, we will likely register more classes in the future so this will only increase.
> The commit added 26 types to register every time we create a Kryo serializer instance. I ran a small microbenchmark to prove that this is noticeably expensive:
> {code}
> import org.apache.spark.serializer._
> import org.apache.spark.SparkConf
> def makeMany(num: Int): Long = {
>   val start = System.currentTimeMillis
>   (1 to num).foreach { _ => new KryoSerializer(new SparkConf).newKryo() }
>   System.currentTimeMillis - start
> }
> // before commit de02782, averaged over multiple runs
> makeMany(5000) == 1500
> // after commit de02782, averaged over multiple runs
> makeMany(5000) == 2750
> {code}
> Since we create multiple serializer instances per partition, this means a 5000-partition stage will unconditionally see an increase of > 1s for the stage. In page rank, we may run many such stages.
> We should explore the alternative of reusing thread-local serializer instances, which would lead to much fewer calls to `kryo.register`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org