You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by Jiadong Wu <jw...@gatech.edu> on 2013/01/16 22:17:24 UTC

File input format in GiraphRunner.java

Hi All,

I'm new to Giraph and related hadoop projects. After playing with it
for several days I finally get the shortest path example running. The
roadblock turns out to be the file input format setting in
GiraphRunner.java.

With the input files provided at wiki page, the shortest path example
should work out of the box. However, the following command will lead
to a IOException in master thread,
$ hadoop jar giraph-0.2-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsVertex -w 3 -if
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-ip shortestPathsInputGraph -of
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexOutputFormat
-op shortestPathsOutputGraph

The reason is that the underlying TextVertexInputFormat reads data
from GiraphTextInputFormat but GiraphRunner sets input path with
hadoop.mapreduce.lib.input.FileInputFormat. To run the shortest path
example, the GiraphRunner.java should import
org.apache.giraph.io.formats.GiraphFileInputFormat and should use
GiraphFileInputFormat.addVertexInputPath to set the path. However, I'm
not sure whether this modification will make GiraphRunner less
general.

Regards,

Jiadong

Re: File input format in GiraphRunner.java

Posted by Alessandro Presta <al...@fb.com>.
Patch is up for review: https://issues.apache.org/jira/browse/GIRAPH-431

You will have to replace "if" with "vif" and "ip" with "vip" in your
command.
Let me know if there are any more issues.

Alessandro

On 1/16/13 2:22 PM, "Alessandro Presta" <al...@fb.com> wrote:

>Hi Jiadong,
>
>You're absolutely right.
>I'll fix this as part of GIRAPH-431 (EdgeInputFormat support in
>GiraphRunner).
>
>Thanks for catching this!
>
>Alessandro
>
>On 1/16/13 1:17 PM, "Jiadong Wu" <jw...@gatech.edu> wrote:
>
>>Hi All,
>>
>>I'm new to Giraph and related hadoop projects. After playing with it
>>for several days I finally get the shortest path example running. The
>>roadblock turns out to be the file input format setting in
>>GiraphRunner.java.
>>
>>With the input files provided at wiki page, the shortest path example
>>should work out of the box. However, the following command will lead
>>to a IOException in master thread,
>>$ hadoop jar 
>>giraph-0.2-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar
>>org.apache.giraph.GiraphRunner
>>org.apache.giraph.examples.SimpleShortestPathsVertex -w 3 -if
>>org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>-ip shortestPathsInputGraph -of
>>org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexOutputFormat
>>-op shortestPathsOutputGraph
>>
>>The reason is that the underlying TextVertexInputFormat reads data
>>from GiraphTextInputFormat but GiraphRunner sets input path with
>>hadoop.mapreduce.lib.input.FileInputFormat. To run the shortest path
>>example, the GiraphRunner.java should import
>>org.apache.giraph.io.formats.GiraphFileInputFormat and should use
>>GiraphFileInputFormat.addVertexInputPath to set the path. However, I'm
>>not sure whether this modification will make GiraphRunner less
>>general.
>>
>>Regards,
>>
>>Jiadong
>

Re: File input format in GiraphRunner.java

Posted by Alessandro Presta <al...@fb.com>.
Hi Jiadong,

You're absolutely right.
I'll fix this as part of GIRAPH-431 (EdgeInputFormat support in
GiraphRunner).

Thanks for catching this!

Alessandro

On 1/16/13 1:17 PM, "Jiadong Wu" <jw...@gatech.edu> wrote:

>Hi All,
>
>I'm new to Giraph and related hadoop projects. After playing with it
>for several days I finally get the shortest path example running. The
>roadblock turns out to be the file input format setting in
>GiraphRunner.java.
>
>With the input files provided at wiki page, the shortest path example
>should work out of the box. However, the following command will lead
>to a IOException in master thread,
>$ hadoop jar 
>giraph-0.2-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar
>org.apache.giraph.GiraphRunner
>org.apache.giraph.examples.SimpleShortestPathsVertex -w 3 -if
>org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>-ip shortestPathsInputGraph -of
>org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexOutputFormat
>-op shortestPathsOutputGraph
>
>The reason is that the underlying TextVertexInputFormat reads data
>from GiraphTextInputFormat but GiraphRunner sets input path with
>hadoop.mapreduce.lib.input.FileInputFormat. To run the shortest path
>example, the GiraphRunner.java should import
>org.apache.giraph.io.formats.GiraphFileInputFormat and should use
>GiraphFileInputFormat.addVertexInputPath to set the path. However, I'm
>not sure whether this modification will make GiraphRunner less
>general.
>
>Regards,
>
>Jiadong