You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Adam Roberts (JIRA)" <ji...@apache.org> on 2016/11/01 14:57:58 UTC
[jira] [Created] (SPARK-18197) Optimise AppendOnlyMap
implementation
Adam Roberts created SPARK-18197:
------------------------------------
Summary: Optimise AppendOnlyMap implementation
Key: SPARK-18197
URL: https://issues.apache.org/jira/browse/SPARK-18197
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.0.1, 1.6.2
Reporter: Adam Roberts
Priority: Minor
This improvement works by using the cheapest comparison test first and we observed a 1% performance on PageRank (HiBench large) with this change
tprof output before the change follows, AppendOnlyMap.changeValue is where the optimisation occurs:
{code}
PID 256059 22.86 java_1fa29
MOD 81337 7.26 JITCODE
SYM 11250 1.00 java/io/ObjectOutputStream.writeObject0(Ljava/lang/Object;Z)V_7fe098983af4
SYM 8053 0.72 org/apache/spark/util/collection/AppendOnlyMap.changeValue(Ljava/lang/Object;Lscala/Function2;)Ljava/lang/Object;_7fe098c211e8
SYM 5175 0.46 java/lang/String.equals(Ljava/lang/Object;)Z_7fe0989eb2e8
SYM 3616 0.32 org/apache/spark/util/SizeEstimator$.estimate(Ljava/lang/Object;Ljava/util/IdentityHashMap;)J_7fe098bc35a8
SYM 3235 0.29 org/apache/spark/util/collection/ExternalSorter$$anonfun$4$$anon$6.compare(Ljava/lang/Object;Ljava/lang/Object;)I_7fe098c855a8
SYM 3182 0.28 java/io/ObjectInputStream$BlockDataInputStream.readUTFBody(J)Ljava/lang/String;_7fe098980ec8
SYM 3111 0.28 org/apache/spark/util/SizeEstimator$SearchState.enqueue(Ljava/lang/Object;)V_7fe0989f2920
{code}
tprof after the change
{code}
MOD 56804 5.07 JITCODE
SYM 8766 0.78 java/io/ObjectOutputStream.writeObject0(Ljava/lang/Object;Z)V_7f0088bb2034
SYM 5746 0.51 java/io/ObjectStreamClass.lookup(Ljava/lang/Class;Z)Ljava/io/ObjectStreamClass;_7f0088944ae8
SYM 3378 0.30 java/io/ObjectInputStream.readObject0(Z)Ljava/lang/Object;_7f0088c5ed00
SYM 3121 0.28 java/io/ObjectInputStream$BlockDataInputStream.readUTFBody(J)Ljava/lang/String;_7f0088c6de08
SYM 2857 0.26 org/apache/spark/storage/BufferReleasingInputStream.read([BII)I_7f0088b3e3a8
SYM 2786 0.25 org/apache/spark/util/collection/AppendOnlyMap.changeValue(Ljava/lang/Object;Lscala/Function2;)Ljava/lang/Object;_7f008899b048
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org