You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2021/05/03 10:44:00 UTC
[jira] [Commented] (SPARK-35251) Improve
LiveEntityHelpers.newAccumulatorInfos performace
[ https://issues.apache.org/jira/browse/SPARK-35251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17338303#comment-17338303 ]
Hyukjin Kwon commented on SPARK-35251:
--------------------------------------
Can you show reproducer and how you managed to get the profile?
> Improve LiveEntityHelpers.newAccumulatorInfos performace
> --------------------------------------------------------
>
> Key: SPARK-35251
> URL: https://issues.apache.org/jira/browse/SPARK-35251
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.2.0
> Reporter: Yuming Wang
> Priority: Major
>
> [It|https://github.com/apache/spark/blob/3854ad87c78f2a331f9c9c1a34f9ec281900f8fe/core/src/main/scala/org/apache/spark/status/LiveEntity.scala#L646-L661] will impact performance if there are lot of {{AccumulableInfo}} instances.
> {noformat}
> num #instances #bytes class name
> ----------------------------------------------
> 1: 331897850 50708371592 [C
> 2: 795358 24591381896 [J
> 3: 330836234 10586759488 java.lang.String
> 4: 139726297 8942483008 org.apache.spark.sql.execution.metric.SQLMetric
> 5: 249040156 5976963744 scala.Some
> 6: 146278719 5851148760 org.apache.spark.util.AccumulatorMetadata
> 7: 38448113 5536528272 java.net.URI
> 8: 37540162 3603855552 org.apache.hadoop.fs.FileStatus
> 9: 69724130 3346758240 java.util.Hashtable$Entry
> 10: 61521559 2953034832 java.util.concurrent.ConcurrentHashMap$Node
> 11: 50421974 2823630544 scala.collection.mutable.LinkedEntry
> 12: 43349222 2774350208 org.apache.spark.scheduler.AccumulableInfo
> {noformat}
> {noformat}
> --- 15430388364 ns (2.03%), 1543 samples
> [ 0] scala.collection.TraversableLike.noneIn$1
> [ 1] scala.collection.TraversableLike.filterImpl
> [ 2] scala.collection.TraversableLike.filterImpl$
> [ 3] scala.collection.AbstractTraversable.filterImpl
> [ 4] scala.collection.TraversableLike.filter
> [ 5] scala.collection.TraversableLike.filter$
> [ 6] scala.collection.AbstractTraversable.filter
> [ 7] org.apache.spark.status.LiveEntityHelpers$.newAccumulatorInfos
> [ 8] org.apache.spark.status.LiveTask.doUpdate
> [ 9] org.apache.spark.status.LiveEntity.write
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org