Run SimpleShortestPathsVertex sample application using multiple workers

[Stefan Beskow <St...@sas.com>]

Hi.

I'm trying to run Giraph on Hadoop 2.0.0-cdh4.2.0 using a cluster with 60 nodes. When I run the sample application org.apache.giraph.examples.SimpleShortestPathsVertex with just 1 worker it works fine, but when I specify more than 1 worker it throws exception java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration as shown below. Is there a way to pass a command line parameter to Giraph so that it doesn't use the local job runner or do I need to update any of the Hadoop configuration files for this to work?

Here is the command I use to run sample application with 2 workers:
hadoop jar giraph-examples-1.0.0-for-hadoop-2.0.0-cdh4.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner -Dgiraph.zkList=rdcgrd001.unx.sas.com:2181 -libjars giraph-examples-1.0.0-for-hadoop-2.0.0-cdh4.2.0-jar-with-dependencies.jar org.apache.giraph.examples.SimpleShortestPathsVertex -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/stbesk/input/tiny_graph.txt -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/stbesk/output/shortestpathsC2 -ca SimpleShortestPathsVertex.source=2 -w 2 -ca giraph.SplitMasterWorker=true

Here is the exception:
14/02/24 00:20:23 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one.
14/02/24 00:20:23 INFO utils.ConfigurationUtils: Setting custom argument [SimpleShortestPathsVertex.source] to [2] in GiraphConfiguration
14/02/24 00:20:23 INFO utils.ConfigurationUtils: Setting custom argument [giraph.SplitMasterWorker] to [true] in GiraphConfiguration
14/02/24 00:20:23 WARN job.GiraphConfigurationValidator: Output format vertex index type is not known
14/02/24 00:20:23 WARN job.GiraphConfigurationValidator: Output format vertex value type is not known
14/02/24 00:20:23 WARN job.GiraphConfigurationValidator: Output format edge value type is not known
14/02/24 00:20:23 INFO job.GiraphJob: run: Since checkpointing is disabled (default), do not allow any task retries (setting mapred.map.max.attempts = 0, old value = 4)
Exception in thread "main" java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration: When using LocalJobRunner, must have only one worker since only 1 task at a time!
        at org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:151)
        at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:225)
        at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

Appreciate any help.

Thanks.
Stefan