You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2016/04/27 22:35:13 UTC
[jira] [Assigned] (SPARK-14966) SizeEstimator should ignore classes
in the scala.reflect package
[ https://issues.apache.org/jira/browse/SPARK-14966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-14966:
------------------------------------
Assignee: Josh Rosen (was: Apache Spark)
> SizeEstimator should ignore classes in the scala.reflect package
> ----------------------------------------------------------------
>
> Key: SPARK-14966
> URL: https://issues.apache.org/jira/browse/SPARK-14966
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Josh Rosen
> Assignee: Josh Rosen
>
> In local benchmarks, I noticed SizeEstimator spending tons of time estimating the size of objects which contain TypeTag or ClassTag fields. The problem with these tags is that they reference global Scala reflection objects, which, in turn, reference many singletons, such as TestHive. This throws off the size estimation and wastes tons of time traversing the huge object graph.
> As a result, I think that SizeEstimator should ignore any classes in the {{scala.reflect}} package.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org