You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2016/01/06 11:06:39 UTC
[jira] [Resolved] (SPARK-4725) Re-think custom shuffle serializers
for vertex messages
[ https://issues.apache.org/jira/browse/SPARK-4725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-4725.
------------------------------
Resolution: Won't Fix
> Re-think custom shuffle serializers for vertex messages
> -------------------------------------------------------
>
> Key: SPARK-4725
> URL: https://issues.apache.org/jira/browse/SPARK-4725
> Project: Spark
> Issue Type: Improvement
> Components: GraphX
> Reporter: Takeshi Yamamuro
> Priority: Minor
>
> These serializers are removed in Spark-3649 because some type mismatch
> errors occur in SortShuffleWriter.
> https://www.mail-archive.com/commits@spark.apache.org/msg04125.html
> However, messages between executors might be of critical performance issues in PageRank and other communication-intensive graph tasks.
> Ankur reported that the removal caused a slowdown and the increase
> of per-iteration communications in the commit log.
> I made a patch to avoid the type-mismatch error in
> https://github.com/maropu/spark/commit/20e74f0e41ed99cb0a89ec5e5fc0e3c9e3f1038e#diff-68f4d319d5a58cbe0729476e0cb8594aR39
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org