You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Qifan Pu (JIRA)" <ji...@apache.org> on 2016/07/24 22:29:20 UTC
[jira] [Created] (SPARK-16699) Fix performance bug in hash
aggregate on long string keys
Qifan Pu created SPARK-16699:
--------------------------------
Summary: Fix performance bug in hash aggregate on long string keys
Key: SPARK-16699
URL: https://issues.apache.org/jira/browse/SPARK-16699
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.0.0
Reporter: Qifan Pu
Fix For: 2.0.0
In the following code in `VectorizedHashMapGenerator.scala`:
```
def hashBytes(b: String): String = {
val hash = ctx.freshName("hash")
val bytes = ctx.freshName("bytes")
s"""
|int $result = 0;
|byte[] $bytes = $b;
|for (int i = 0; i < $bytes.length; i++) {
| ${genComputeHash(ctx, s"$bytes[i]", ByteType, hash)}
| $result = ($result ^ (0x9e3779b9)) + $hash + ($result << 6) + ($result >>> 2);
|}
""".stripMargin
}
```
when b=input.getBytes(), the current 2.0 code results in getBytes() being called n times, n being length of input. getBytes() involves memory copy is thus expensive and causes a performance degradation.
Fix is to evaluate getBytes() before the for loop.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org