You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by Armando <a....@student.vu.nl> on 2013/09/10 22:14:15 UTC
Review Request 14068: GIRAPH-723: Rexster Output Format
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14068/
-----------------------------------------------------------
Review request for giraph.
Repository: giraph-git
Description
-------
This patch provides a way to use Rexster as output format endpoint. By means of ZooKeeper coordination, the workers will first write all the vertices and then all the edges. Such an order is required by the bluprint API.
Limitations:
- The current approach does not yet work with neo4j. In fact, neo4j does not properly handle saving vertices not in order of ID. I am looking into this currently.
- This approach does not scale. In fact, the batch-kibble provided by Rexster and used by the output format loads the whole graph in memory before committing the data collected into the underlining database. This will be solved with a Giraph provided kibble to stream the data from the output format.
Diffs
-----
giraph-rexster/src/main/java/org/apache/giraph/rexster/conf/GiraphRexsterConstants.java c4b4655
giraph-rexster/src/main/java/org/apache/giraph/rexster/io/RexsterEdgeInputFormat.java 4dee078
giraph-rexster/src/main/java/org/apache/giraph/rexster/io/RexsterEdgeOutputFormat.java PRE-CREATION
giraph-rexster/src/main/java/org/apache/giraph/rexster/io/RexsterVertexInputFormat.java da45b5b
giraph-rexster/src/main/java/org/apache/giraph/rexster/io/RexsterVertexOutputFormat.java PRE-CREATION
giraph-rexster/src/main/java/org/apache/giraph/rexster/io/formats/RexsterLongDoubleFloatEdgeOutputFormat.java PRE-CREATION
giraph-rexster/src/main/java/org/apache/giraph/rexster/io/formats/RexsterLongDoubleFloatVertexOutputFormat.java PRE-CREATION
giraph-rexster/src/main/java/org/apache/giraph/rexster/utils/RexsterUtils.java e669ca9
giraph-rexster/src/test/java/org/apache/giraph/rexster/io/formats/TestAbstractRexsterInputFormat.java 09b27d3
giraph-rexster/src/test/java/org/apache/giraph/rexster/io/formats/TestRexsterLongDoubleFloatInputFormat.java PRE-CREATION
giraph-rexster/src/test/java/org/apache/giraph/rexster/io/formats/TestRexsterLongDoubleFloatOutputFormat.java PRE-CREATION
giraph-rexster/src/test/java/org/apache/giraph/rexster/io/formats/TestRexsterLongDoubleFloatVertexInputFormat.java a68db67
Diff: https://reviews.apache.org/r/14068/diff/
Testing
-------
tested on a small cluster using tinkergraph as database exported with Rexster.
Thanks,
Armando
Re: Review Request 14068: GIRAPH-723: Rexster Output Format
Posted by Armando <a....@student.vu.nl>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14068/
-----------------------------------------------------------
(Updated Sept. 15, 2013, 11:53 a.m.)
Review request for giraph.
Changes
-------
add check for the presence of Vertex and Edge formats (both must be there).
Repository: giraph-git
Description
-------
This patch provides a way to use Rexster as output format endpoint. By means of ZooKeeper coordination, the workers will first write all the vertices and then all the edges. Such an order is required by the bluprint API.
Limitations:
- The current approach does not yet work with neo4j. In fact, neo4j does not properly handle saving vertices not in order of ID. I am looking into this currently.
- This approach does not scale. In fact, the batch-kibble provided by Rexster and used by the output format loads the whole graph in memory before committing the data collected into the underlining database. This will be solved with a Giraph provided kibble to stream the data from the output format.
Diffs (updated)
-----
giraph-rexster/src/main/java/org/apache/giraph/rexster/conf/GiraphRexsterConstants.java c4b4655
giraph-rexster/src/main/java/org/apache/giraph/rexster/io/RexsterEdgeInputFormat.java 4dee078
giraph-rexster/src/main/java/org/apache/giraph/rexster/io/RexsterEdgeOutputFormat.java PRE-CREATION
giraph-rexster/src/main/java/org/apache/giraph/rexster/io/RexsterVertexInputFormat.java da45b5b
giraph-rexster/src/main/java/org/apache/giraph/rexster/io/RexsterVertexOutputFormat.java PRE-CREATION
giraph-rexster/src/main/java/org/apache/giraph/rexster/io/formats/RexsterLongDoubleFloatEdgeOutputFormat.java PRE-CREATION
giraph-rexster/src/main/java/org/apache/giraph/rexster/io/formats/RexsterLongDoubleFloatVertexOutputFormat.java PRE-CREATION
giraph-rexster/src/main/java/org/apache/giraph/rexster/utils/RexsterUtils.java e669ca9
giraph-rexster/src/test/java/org/apache/giraph/rexster/io/formats/TestAbstractRexsterInputFormat.java 09b27d3
giraph-rexster/src/test/java/org/apache/giraph/rexster/io/formats/TestRexsterLongDoubleFloatInputFormat.java PRE-CREATION
giraph-rexster/src/test/java/org/apache/giraph/rexster/io/formats/TestRexsterLongDoubleFloatOutputFormat.java PRE-CREATION
giraph-rexster/src/test/java/org/apache/giraph/rexster/io/formats/TestRexsterLongDoubleFloatVertexInputFormat.java a68db67
Diff: https://reviews.apache.org/r/14068/diff/
Testing
-------
tested on a small cluster using tinkergraph as database exported with Rexster.
Thanks,
Armando