Re: GraphX graph partitioning strategy

[Larry Xiao <xi...@sjtu.edu.cn>]

Hi Ankur, all,

I've implemented few graph partitioning algorithms, and done some 
evaluation.
The goal is to lower replication factor and produce better balanced 
graph, so to make work load more balance.
Detailed description and result: 
https://issues.apache.org/jira/browse/SPARK-3523

Can you help take a look?
Thank you!

Larry

On 7/24/14 2:59 PM, Larry Xiao wrote:
> Hi all,
>
> I'm implementing graph partitioning strategy for GraphX, learning from 
> researches on graph computing.
>
> I have two questions:
>
> - a specific implement question:
> In current design, only vertex ID of src and dst are provided 
> (PartitionStrategy.scala).
> And some strategies require knowledge about the graph (like degrees) 
> and can consist more than one passes to finally produce the partition ID.
> So I'm changing the PartitionStrategy.getPartition API to provide more 
> info, but I don't want to make it complex. (the current one looks very 
> clean)
>
> - an open question:
> What advice would you give considering partitioning, considering the 
> procedure Spark adopt on graph processing?
>
> Any advice is much appreciated.
>
> Best Regards,
> Larry Xiao
>
> Reference
>
> Bipartite-oriented Distributed Graph Partitioning for Big Learning.
> PowerLyra : Differentiated Graph Computation and Partitioning on 
> Skewed Graphs
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org