[TinkerPop3] Hadoop-Gremlin now has SparkGraphComputer

[Marko Rodriguez <ok...@gmail.com>]

Hello,

Last week I was at a DataStax company retreat. As such, I didn't have the focus nor time to code. When I found available time here and there, I played with Spark via the scala> console. Stupid simple "word count"-style computations -- just to get my feet wet. However, I figured out how to do the "Faunus-style" MapReduce-based message passing via Spark's functional DSL and so this Monday morning I went to work.

	Day 1 (Monday): By the end of the day, I had hardcoded PageRankVertexProgram working over SparkGraphComputer.
	Day 2 (Tuesday): By the end of the day, I had TraversalVertexProgram (i.e. Gremlin OLAP) working over SparkGraphComputer.
	Day 3 (Wednesday): This morning I got all the integration tests (save 3 !?) working over SparkGraphComputer.

So, to toot the proverbial self-indulgent horn, the above sequence of events is a testament to three things:

	1. Spark is a really easy framework to work with.
	2. The OLAP GraphComputer API of TinkerPop3 is flexible and easy to implement.
	3. I am a ravenous monster on the code base when I have a mission.

This work is currently in master/ and will be available in TP3.M8. You can read the docs here:

	http://www.tinkerpop.com/docs/3.0.0-SNAPSHOT/#_olap_hadoop_gremlin (Hadoop supports 3 GraphComputers)
		http://www.tinkerpop.com/docs/3.0.0-SNAPSHOT/#sparkgraphcomputer (SparkGraphComputer)

Finally, for those interested in the implementation, please see:
	https://github.com/apache/incubator-tinkerpop/tree/master/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/process/computer/spark

The core message passing algorithm is here:
	https://github.com/apache/incubator-tinkerpop/blob/master/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/process/computer/spark/util/SparkHelper.java#L58

Note that I am no expert in Spark. As such, over time I hope to learn more (I bought a Spark book on Amazon.com) and will iterate to make the message passing algorithm both more time and space efficient.

Enjoy!,
Marko.

http://markorodriguez.com