Giraph insists on LocalJobRunner with custom Computation

[Yørn de Jong <yo...@uninett.no>]

Hi group

I have set up Giraph on a YARN cluster. I have no trouble running the shortest paths example as described in [1], but when I try to run my own algorithm, the program stops with:

    Exception in thread "main" Java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration: When using LocalJobRunner, must have only one worker since only 1 task at a time!

When I change the command to -w 1, it stops with

    Exception in thread "main" java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run in split master / worker mode since there is only 1 task at a time!

The command I try to run is

    hadoop \
    jar giraph-rank-1.1.0-SNAPSHOT-for-hadoop-2.3.0-cdh5.0.1-jar-with-dependencies.jar \
    org.apache.giraph.GiraphRunner no.uninett.yorn.giraph.computation.DOSRank \
    -eif no.uninett.yorn.giraph.format.io.NetflowCSVEdgeInputFormat \
    -eip /user/hdfs/trd_gw1_12_01_normalized.csv \
    -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat \
    -op /user/yarn/output \
    -wc org.apache.giraph.worker.DefaultWorkerContext \
    -w 5 \
    -yj giraph-rank-1.1.0-SNAPSHOT-for-hadoop-2.3.0-cdh5.0.1-jar-with-dependencies.jar

Where, naturally, no.uninett.yorn.giraph.computation.DOSRank is my own algorithm, which is contained in giraph-rank-1.1.0-SNAPSHOT-for-hadoop-2.3.0-cdh5.0.1-jar-with-dependencies.jar.

giraph-rank is built using the same command I used to build the giraph-examples project, and the pom.xml for giraph-rank is made by copying from giraph-examples, and replacing «examples» with «rank». The command used to build both projects is:

    mvn -Phadoop_yarn -Dhadoop.version=2.3.0-cdh5.0.1 clean package -DskipTests

Interestingly enough, when I change the input format to something else, I get error messages having to do with type mismatches. This seems to suggest that my EdgeInputFormat does start, but that the problem occurs while it runs or after it runs. I don’t know how to debug this.

Am I missing something? How can I get my algorithm to run?
The source code is available on [2].

[1] https://giraph.apache.org/quick_start.html#qs_section_5
[2] https://scm.uninett.no/yorn/giraph

Re: Giraph insists on LocalJobRunner with custom Computation

[John Yost <so...@gmail.com>]

Hi Yorn,

I figured this out and detailed the solution in my post earlier this
morning (6/23 2:46). The key is the following: -ca
mapred.job.tracker=localhost:5431.  Without this, you'll see the exception
you detailed above.

--John


On Thu, Jun 5, 2014 at 5:36 AM, Yørn de Jong <yo...@uninett.no> wrote:

> Hi group
>
> I have set up Giraph on a YARN cluster. I have no trouble running the
> shortest paths example as described in [1], but when I try to run my own
> algorithm, the program stops with:
>
>     Exception in thread "main" Java.lang.IllegalArgumentException:
> checkLocalJobRunnerConfiguration: When using LocalJobRunner, must have only
> one worker since only 1 task at a time!
>
> When I change the command to -w 1, it stops with
>
>     Exception in thread "main" java.lang.IllegalArgumentException:
> checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run
> in split master / worker mode since there is only 1 task at a time!
>
> The command I try to run is
>
>     hadoop \
>     jar
> giraph-rank-1.1.0-SNAPSHOT-for-hadoop-2.3.0-cdh5.0.1-jar-with-dependencies.jar
> \
>     org.apache.giraph.GiraphRunner
> no.uninett.yorn.giraph.computation.DOSRank \
>     -eif no.uninett.yorn.giraph.format.io.NetflowCSVEdgeInputFormat \
>     -eip /user/hdfs/trd_gw1_12_01_normalized.csv \
>     -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat \
>     -op /user/yarn/output \
>     -wc org.apache.giraph.worker.DefaultWorkerContext \
>     -w 5 \
>     -yj
> giraph-rank-1.1.0-SNAPSHOT-for-hadoop-2.3.0-cdh5.0.1-jar-with-dependencies.jar
>
> Where, naturally, no.uninett.yorn.giraph.computation.DOSRank is my own
> algorithm, which is contained in
> giraph-rank-1.1.0-SNAPSHOT-for-hadoop-2.3.0-cdh5.0.1-jar-with-dependencies.jar.
>
> giraph-rank is built using the same command I used to build the
> giraph-examples project, and the pom.xml for giraph-rank is made by copying
> from giraph-examples, and replacing «examples» with «rank». The command
> used to build both projects is:
>
>     mvn -Phadoop_yarn -Dhadoop.version=2.3.0-cdh5.0.1 clean package
> -DskipTests
>
> Interestingly enough, when I change the input format to something else, I
> get error messages having to do with type mismatches. This seems to suggest
> that my EdgeInputFormat does start, but that the problem occurs while it
> runs or after it runs. I don’t know how to debug this.
>
> Am I missing something? How can I get my algorithm to run?
> The source code is available on [2].
>
> [1] https://giraph.apache.org/quick_start.html#qs_section_5
> [2] https://scm.uninett.no/yorn/giraph
>