GraphX: how to specify partition strategy?

[Yifan LI <ia...@gmail.com>]

Hi,

I am doing graph computation using GraphX, but it seems to be an error on graph partition strategy specification.
as in "GraphX programming guide":
The Graph.partitionBy operator allows users to choose the graph partitioning strategy, but due to SPARK-1931, this method is broken in Spark 1.0.0. We encourage users to build the latest version of Spark from the master branch, which contains a fix. Alternatively, a workaround is to partition the edges before constructing the graph, as follows:

My questions:

- how to "build the latest version of Spark from the master branch, which contains a fix"?

- how to specify other partition strategy, eg. CanonicalRandomVertexCut, EdgePartition1D, EdgePartition2D, RandomVertexCut
(listed in Scala API document, but seems only "EdgePartition2D" is available? I am not sure for this! )

- Is it possible to add my own partition strategy(hash function, etc.) into GraphX?

Thanks in advance! :))

Best,
Yifan

Re: GraphX: how to specify partition strategy?

[Ankur Dave <an...@gmail.com>]

On Thu, Jul 10, 2014 at 8:20 AM, Yifan LI <ia...@gmail.com> wrote:
>
> - how to "build the latest version of Spark from the master branch, which
> contains a fix"?


Instead of downloading a prebuilt Spark release from
http://spark.apache.org/downloads.html, follow the instructions under
"Development Version" on that page. In short:

git clone git://github.com/apache/spark.git
cd spark
sbt/sbt assembly

Then you can run bin/spark-shell and bin/spark-submit as usual, and
Graph.partitionBy should work.

- how to specify other partition strategy, eg. CanonicalRandomVertexCut,
> EdgePartition1D, EdgePartition2D, RandomVertexCut
> (listed in Scala API document, but seems only "EdgePartition2D" is
> available? I am not sure for this! )


All of those partition strategies should be available -- for example, you
can call graph.partitionBy(PartitionStrategy.RandomVertexCut).

- Is it possible to add my own partition strategy(hash function, etc.) into
> GraphX?


Yes, you just need to create a subclass of PartitionStrategy as follows:

import org.apache.spark.graphx._

object MyPartitionStrategy extends PartitionStrategy {
  override def getPartition(src: VertexId, dst: VertexId, numParts:
PartitionID): PartitionID = {
    // put your hash function here
  }
}

Ankur <http://www.ankurdave.com/>