You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Reynold Xin (JIRA)" <ji...@apache.org> on 2014/11/26 06:44:12 UTC
[jira] [Commented] (SPARK-3936) Remove auto join elimination and
introduce aggregateMessages
[ https://issues.apache.org/jira/browse/SPARK-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225788#comment-14225788 ]
Reynold Xin commented on SPARK-3936:
------------------------------------
BTW message from Taobao: 根据团队小伙伴的测试，[SPARK-3936] Add aggregateMessages, which supersedes mapReduceTriplets这个issue确实Work，可以提升30%的性能。以KCore为例，在standord的100w的数据集上运行作业，运行时间由 37mins减到22mins。使用GraphX的同学可以早日升级。#给周末此时还在加班做Spark和图计算的同学#
i.e. switching from mapReduceTriplets to aggregateMessages improved performance by 30% in their kcore implementation.
> Remove auto join elimination and introduce aggregateMessages
> ------------------------------------------------------------
>
> Key: SPARK-3936
> URL: https://issues.apache.org/jira/browse/SPARK-3936
> Project: Spark
> Issue Type: Bug
> Components: GraphX
> Reporter: Pedro Rodriguez
> Assignee: Ankur Dave
> Priority: Blocker
> Fix For: 1.2.0
>
>
> This is actually a ticket with two separate problems:
> 1. Remove auto join elimination
> 2. Introduce a new fundamental primitive aggregateMessages
> For the first one, description provided by Pedro:
> There seems to be a bug with the GraphX byte code inspection, specifically in BytecodeUtils.
> These are the unit tests I wrote to expose the problem:
> https://github.com/EntilZha/spark/blob/a3c38a8329545c034fae2458df134fa3829d08fb/graphx/src/test/scala/org/apache/spark/graphx/util/BytecodeUtilsSuite.scala#L93-L121
> The first two tests pass, the second two tests fail. This exposes a problem with inspection of methods in closures, in this case within maps. Specifically, it seems like there is a problem with inspection of non-inline methods in a closure.
> For the 2nd one, see pull request https://github.com/apache/spark/pull/3100
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org