You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Kenrick Fernandes <ke...@gmail.com> on 2015/03/22 21:28:13 UTC
Giraph 1.1.0 not running on full cluster with Hadoop 2.6.0
Hi,
I am working with Giraph 1.1.0 and a YARN cluster with Hadoop 2.6.0.
I build Giraph with
mvn -Phadoop_2 -Dhadoop.version=2.6.0 clean package -DskipTests
So far, when I run any of the benchmarks or Shortest path examples, I
always get the LocalJobRunner error :
---------------------------------------------------------------------------------------
Exception in thread "main" java.lang.IllegalArgumentException:
checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run
in split master / worker mode since there is only 1 task at a time!
at
org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:168)
at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:236)
at org.apache.giraph.benchmark.GiraphBenchmark.run(GiraphBenchmark.java:96)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.giraph.benchmark.PageRankBenchmark.main(PageRankBenchmark.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
-------------------------------------------------------------------
I have tried some solutions from forums/StackOverflow/lists, but so far
nothing has worked. As far as I can tell, Hadoop is configured right (other
MR benchmarks run fine). I tried changing the above Giraph code file
(making the check function returned what I wanted), but that only starts
the Job and gets it running on a single machine - it never uses more than 1
machine.
Any help or pointers in the right direction would be much appreciated.
Thanks,
Kenrick
Re: Giraph 1.1.0 not running on full cluster with Hadoop 2.6.0
Posted by Kenrick Fernandes <ke...@gmail.com>.
Update:
The shortest path example runs now, but only with *1 worker*. Using more
than 1 worker gives the same
*LocalJobRunner *error:
-----------------------------------------
*Command:*
hadoop jar giraph-examples-1.1.0-for-hadoop-2.6.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation
-vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /user/kenrick/input/tiny_graph.txt
-vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat
-op /user/kenrick/output/shortestpaths -w 2
-ca giraph.SplitMasterWorker=false
*Error:*
Exception in thread "main" java.lang.IllegalArgumentException:
checkLocalJobRunnerConfiguration: When using LocalJobRunner, must have only
one worker since only 1 task at a time!
at
org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:162)
at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:236)
at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
-----------------------------------------
Suggestions/fixes would be much appreciated.
Thanks,
Kenrick
On Mon, Mar 23, 2015 at 1:30 PM, Kenrick Fernandes <ke...@gmail.com>
wrote:
> Hi Steve,
>
> Running the Shortest Paths example with that option gives me an
> *IncompatibleClassChange* error, which
> I am still trying to figure out how to fix. However, running the PageRank
> benchmark gives an *Unrecognized option*
> error (both are below) - is there a specific way to pass the param
> depending on whether GiraphRunner is present?
>
>
> -----------------------------------------
> *Command:*
> hadoop jar
> giraph-examples-1.1.0-for-hadoop-2.6.0-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation
> -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip /user/kenrick/input/tiny_graph.txt
> -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat
> -op /user/kenrick/output/shortestpaths -w 1 -ca
> giraph.SplitMasterWorker=false
>
> *Error:*
> Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
> interface org.apache.hadoop.mapreduce.JobContext, but class was expected
> at
> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:562)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
>
> -----------------------------------------
> *Command:*
> hadoop jar giraph-1.1.0-for-hadoop-2.6.0-jar-with-dependencies.jar
> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000 -w
> 30 -ca giraph.SplitMasterWorker=false
>
> *Error:*
> Exception in thread "main"
> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: -ca
> at org.apache.commons.cli.Parser.processOption(Parser.java:363)
> at org.apache.commons.cli.Parser.parse(Parser.java:199)
> at org.apache.commons.cli.Parser.parse(Parser.java:85)
> at org.apache.giraph.benchmark.GiraphBenchmark.run(GiraphBenchmark.java:71)
>
> -----------------------------------------
>
> Thanks,
> Kenrick
>
> On Mon, Mar 23, 2015 at 9:26 AM, Steven Harenberg <sd...@ncsu.edu>
> wrote:
>
>> Try adding this option: -ca giraph.SplitMasterWorker=false
>>
>> --Steve
>>
>>
>> On Sun, Mar 22, 2015 at 4:28 PM, Kenrick Fernandes <kenrick.f15@gmail.com
>> > wrote:
>>
>>> Hi,
>>>
>>> I am working with Giraph 1.1.0 and a YARN cluster with Hadoop 2.6.0.
>>> I build Giraph with
>>> mvn -Phadoop_2 -Dhadoop.version=2.6.0 clean package -DskipTests
>>>
>>> So far, when I run any of the benchmarks or Shortest path examples, I
>>> always get the LocalJobRunner error :
>>>
>>>
>>>
>>> ---------------------------------------------------------------------------------------
>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>> checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run
>>> in split master / worker mode since there is only 1 task at a time!
>>> at
>>> org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:168)
>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:236)
>>> at
>>> org.apache.giraph.benchmark.GiraphBenchmark.run(GiraphBenchmark.java:96)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>> at
>>> org.apache.giraph.benchmark.PageRankBenchmark.main(PageRankBenchmark.java:158)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>>> -------------------------------------------------------------------
>>>
>>> I have tried some solutions from forums/StackOverflow/lists, but so far
>>> nothing has worked. As far as I can tell, Hadoop is configured right (other
>>> MR benchmarks run fine). I tried changing the above Giraph code file
>>> (making the check function returned what I wanted), but that only starts
>>> the Job and gets it running on a single machine - it never uses more than 1
>>> machine.
>>>
>>> Any help or pointers in the right direction would be much appreciated.
>>>
>>> Thanks,
>>> Kenrick
>>>
>>
>>
>
Re: Giraph 1.1.0 not running on full cluster with Hadoop 2.6.0
Posted by Kenrick Fernandes <ke...@gmail.com>.
Hi Steve,
Running the Shortest Paths example with that option gives me an
*IncompatibleClassChange* error, which
I am still trying to figure out how to fix. However, running the PageRank
benchmark gives an *Unrecognized option*
error (both are below) - is there a specific way to pass the param
depending on whether GiraphRunner is present?
-----------------------------------------
*Command:*
hadoop jar giraph-examples-1.1.0-for-hadoop-2.6.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation
-vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /user/kenrick/input/tiny_graph.txt
-vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat
-op /user/kenrick/output/shortestpaths -w 1 -ca
giraph.SplitMasterWorker=false
*Error:*
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
interface org.apache.hadoop.mapreduce.JobContext, but class was expected
at
org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
at
org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:562)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
-----------------------------------------
*Command:*
hadoop jar giraph-1.1.0-for-hadoop-2.6.0-jar-with-dependencies.jar
org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000 -w
30 -ca giraph.SplitMasterWorker=false
*Error:*
Exception in thread "main"
org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: -ca
at org.apache.commons.cli.Parser.processOption(Parser.java:363)
at org.apache.commons.cli.Parser.parse(Parser.java:199)
at org.apache.commons.cli.Parser.parse(Parser.java:85)
at org.apache.giraph.benchmark.GiraphBenchmark.run(GiraphBenchmark.java:71)
-----------------------------------------
Thanks,
Kenrick
On Mon, Mar 23, 2015 at 9:26 AM, Steven Harenberg <sd...@ncsu.edu> wrote:
> Try adding this option: -ca giraph.SplitMasterWorker=false
>
> --Steve
>
>
> On Sun, Mar 22, 2015 at 4:28 PM, Kenrick Fernandes <ke...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am working with Giraph 1.1.0 and a YARN cluster with Hadoop 2.6.0.
>> I build Giraph with
>> mvn -Phadoop_2 -Dhadoop.version=2.6.0 clean package -DskipTests
>>
>> So far, when I run any of the benchmarks or Shortest path examples, I
>> always get the LocalJobRunner error :
>>
>>
>>
>> ---------------------------------------------------------------------------------------
>> Exception in thread "main" java.lang.IllegalArgumentException:
>> checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run
>> in split master / worker mode since there is only 1 task at a time!
>> at
>> org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:168)
>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:236)
>> at
>> org.apache.giraph.benchmark.GiraphBenchmark.run(GiraphBenchmark.java:96)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> at
>> org.apache.giraph.benchmark.PageRankBenchmark.main(PageRankBenchmark.java:158)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>> -------------------------------------------------------------------
>>
>> I have tried some solutions from forums/StackOverflow/lists, but so far
>> nothing has worked. As far as I can tell, Hadoop is configured right (other
>> MR benchmarks run fine). I tried changing the above Giraph code file
>> (making the check function returned what I wanted), but that only starts
>> the Job and gets it running on a single machine - it never uses more than 1
>> machine.
>>
>> Any help or pointers in the right direction would be much appreciated.
>>
>> Thanks,
>> Kenrick
>>
>
>
Re: Giraph 1.1.0 not running on full cluster with Hadoop 2.6.0
Posted by Steven Harenberg <sd...@ncsu.edu>.
Try adding this option: -ca giraph.SplitMasterWorker=false
--Steve
On Sun, Mar 22, 2015 at 4:28 PM, Kenrick Fernandes <ke...@gmail.com>
wrote:
> Hi,
>
> I am working with Giraph 1.1.0 and a YARN cluster with Hadoop 2.6.0.
> I build Giraph with
> mvn -Phadoop_2 -Dhadoop.version=2.6.0 clean package -DskipTests
>
> So far, when I run any of the benchmarks or Shortest path examples, I
> always get the LocalJobRunner error :
>
>
>
> ---------------------------------------------------------------------------------------
> Exception in thread "main" java.lang.IllegalArgumentException:
> checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run
> in split master / worker mode since there is only 1 task at a time!
> at
> org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:168)
> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:236)
> at org.apache.giraph.benchmark.GiraphBenchmark.run(GiraphBenchmark.java:96)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.giraph.benchmark.PageRankBenchmark.main(PageRankBenchmark.java:158)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> -------------------------------------------------------------------
>
> I have tried some solutions from forums/StackOverflow/lists, but so far
> nothing has worked. As far as I can tell, Hadoop is configured right (other
> MR benchmarks run fine). I tried changing the above Giraph code file
> (making the check function returned what I wanted), but that only starts
> the Job and gets it running on a single machine - it never uses more than 1
> machine.
>
> Any help or pointers in the right direction would be much appreciated.
>
> Thanks,
> Kenrick
>
Re: Giraph 1.1.0 not running on full cluster with Hadoop 2.6.0
Posted by Steven Harenberg <sd...@ncsu.edu>.
It doesn't work for me, but it seems to be because I am only on a single
machine while it requires split master/worker mode. If I move onto
distributed soon I will let you know.
--Steve
On Wed, Mar 25, 2015 at 2:56 PM, Kenrick Fernandes <ke...@gmail.com>
wrote:
> Hi Steve,
>
> Thanks for the link - there's a different error I get now regarding not
> finding some other classes,
> but ive seen that before and should be able to find a fix.
>
> Running PageRank however, still gives me the *localJobRunner* error
> (above) - did you get that to run
> successfully?
>
> Thanks,
> Kenrick
>
> On Tue, Mar 24, 2015 at 3:06 PM, Steven Harenberg <sd...@ncsu.edu>
> wrote:
>
>> Hey Kenrick,
>>
>> For the issue with GiraphApplicationMaster, I followed what Phillip did
>> here:
>> http://mail-archives.apache.org/mod_mbox/giraph-user/201503.mbox/%3CCAO3ErG_obGV8mELzX1j%2Be%3DaL6C%3D6%3DtdiSOVRBia2gh0H9tYLZA%40mail.gmail.com%3E
>>
>> Basically you need the jar for giraph-examples to be in the directory
>> where you are issuing the command. You can do this by creating a symbolic
>> link. I have no idea why this worked and you can't use an absolute path,
>> but that is how it was for me.
>>
>> Thanks,
>> Steve
>>
>> On Mon, Mar 23, 2015 at 7:06 PM, Kenrick Fernandes <kenrick.f15@gmail.com
>> > wrote:
>>
>>> Hi Phil,
>>>
>>> The build was successful - now running the *ShortestPaths* example
>>> gives me a different error,
>>> *"GiraphApplicationMaster* not found" . However, when I run the
>>> PageRank benchmark, I still
>>> get the same *LocalJobRunner* error:
>>>
>>> -------------------------
>>> *Command:*
>>> hadoop jar giraph-1.1.0-for-hadoop-2.6.0-jar-with-dependencies.jar
>>> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000 -w 30
>>>
>>> *Error:*
>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>> checkLocalJobRunnerConfiguration: When using LocalJobRunner, must have only
>>> one worker since only 1 task at a time!
>>> at
>>> org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:162)
>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:236)
>>> at
>>> org.apache.giraph.benchmark.GiraphBenchmark.run(GiraphBenchmark.java:96)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>> at
>>> org.apache.giraph.benchmark.PageRankBenchmark.main(PageRankBenchmark.java:158)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>>> -------------------------
>>>
>>> Did PageRank run fine for you ?
>>>
>>> Thanks,
>>> Kenrick
>>>
>>> On Mon, Mar 23, 2015 at 4:35 PM, Phillip Rhodes <
>>> motley.crue.fan@gmail.com> wrote:
>>>
>>>> What I had to do to get this to work was:
>>>>
>>>> edit the pom.xml and change the hadoop_yarn profile to remove the one
>>>> munge symbol that was something like _SASL_SOMETHING_OR_OTHER.
>>>>
>>>> Build using mvn -Phadoop_yarn -Dhadoop.version=2.5.2 (in my case)
>>>>
>>>>
>>>> Phil
>>>>
>>>> This message optimized for indexing by NSA PRISM
>>>>
>>>>
>>>> On Sun, Mar 22, 2015 at 4:28 PM, Kenrick Fernandes
>>>> <ke...@gmail.com> wrote:
>>>> > Hi,
>>>> >
>>>> > I am working with Giraph 1.1.0 and a YARN cluster with Hadoop 2.6.0.
>>>> > I build Giraph with
>>>> > mvn -Phadoop_2 -Dhadoop.version=2.6.0 clean package -DskipTests
>>>> >
>>>> > So far, when I run any of the benchmarks or Shortest path examples, I
>>>> always
>>>> > get the LocalJobRunner error :
>>>> >
>>>> >
>>>> >
>>>> ---------------------------------------------------------------------------------------
>>>> > Exception in thread "main" java.lang.IllegalArgumentException:
>>>> > checkLocalJobRunnerConfiguration: When using LocalJobRunner, you
>>>> cannot run
>>>> > in split master / worker mode since there is only 1 task at a time!
>>>> > at
>>>> >
>>>> org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:168)
>>>> > at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:236)
>>>> > at
>>>> org.apache.giraph.benchmark.GiraphBenchmark.run(GiraphBenchmark.java:96)
>>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>> > at
>>>> >
>>>> org.apache.giraph.benchmark.PageRankBenchmark.main(PageRankBenchmark.java:158)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> > at java.lang.reflect.Method.invoke(Method.java:606)
>>>> > at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>>>> > at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>>>> > -------------------------------------------------------------------
>>>> >
>>>> > I have tried some solutions from forums/StackOverflow/lists, but so
>>>> far
>>>> > nothing has worked. As far as I can tell, Hadoop is configured right
>>>> (other
>>>> > MR benchmarks run fine). I tried changing the above Giraph code file
>>>> (making
>>>> > the check function returned what I wanted), but that only starts the
>>>> Job and
>>>> > gets it running on a single machine - it never uses more than 1
>>>> machine.
>>>> >
>>>> > Any help or pointers in the right direction would be much appreciated.
>>>> >
>>>> > Thanks,
>>>> > Kenrick
>>>>
>>>
>>>
>>
>
Re: Giraph 1.1.0 not running on full cluster with Hadoop 2.6.0
Posted by Kenrick Fernandes <ke...@gmail.com>.
Hi Steve,
Thanks for the link - there's a different error I get now regarding not
finding some other classes,
but ive seen that before and should be able to find a fix.
Running PageRank however, still gives me the *localJobRunner* error (above)
- did you get that to run
successfully?
Thanks,
Kenrick
On Tue, Mar 24, 2015 at 3:06 PM, Steven Harenberg <sd...@ncsu.edu> wrote:
> Hey Kenrick,
>
> For the issue with GiraphApplicationMaster, I followed what Phillip did
> here:
> http://mail-archives.apache.org/mod_mbox/giraph-user/201503.mbox/%3CCAO3ErG_obGV8mELzX1j%2Be%3DaL6C%3D6%3DtdiSOVRBia2gh0H9tYLZA%40mail.gmail.com%3E
>
> Basically you need the jar for giraph-examples to be in the directory
> where you are issuing the command. You can do this by creating a symbolic
> link. I have no idea why this worked and you can't use an absolute path,
> but that is how it was for me.
>
> Thanks,
> Steve
>
> On Mon, Mar 23, 2015 at 7:06 PM, Kenrick Fernandes <ke...@gmail.com>
> wrote:
>
>> Hi Phil,
>>
>> The build was successful - now running the *ShortestPaths* example gives
>> me a different error,
>> *"GiraphApplicationMaster* not found" . However, when I run the PageRank
>> benchmark, I still
>> get the same *LocalJobRunner* error:
>>
>> -------------------------
>> *Command:*
>> hadoop jar giraph-1.1.0-for-hadoop-2.6.0-jar-with-dependencies.jar
>> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000 -w 30
>>
>> *Error:*
>> Exception in thread "main" java.lang.IllegalArgumentException:
>> checkLocalJobRunnerConfiguration: When using LocalJobRunner, must have only
>> one worker since only 1 task at a time!
>> at
>> org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:162)
>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:236)
>> at
>> org.apache.giraph.benchmark.GiraphBenchmark.run(GiraphBenchmark.java:96)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> at
>> org.apache.giraph.benchmark.PageRankBenchmark.main(PageRankBenchmark.java:158)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>> -------------------------
>>
>> Did PageRank run fine for you ?
>>
>> Thanks,
>> Kenrick
>>
>> On Mon, Mar 23, 2015 at 4:35 PM, Phillip Rhodes <
>> motley.crue.fan@gmail.com> wrote:
>>
>>> What I had to do to get this to work was:
>>>
>>> edit the pom.xml and change the hadoop_yarn profile to remove the one
>>> munge symbol that was something like _SASL_SOMETHING_OR_OTHER.
>>>
>>> Build using mvn -Phadoop_yarn -Dhadoop.version=2.5.2 (in my case)
>>>
>>>
>>> Phil
>>>
>>> This message optimized for indexing by NSA PRISM
>>>
>>>
>>> On Sun, Mar 22, 2015 at 4:28 PM, Kenrick Fernandes
>>> <ke...@gmail.com> wrote:
>>> > Hi,
>>> >
>>> > I am working with Giraph 1.1.0 and a YARN cluster with Hadoop 2.6.0.
>>> > I build Giraph with
>>> > mvn -Phadoop_2 -Dhadoop.version=2.6.0 clean package -DskipTests
>>> >
>>> > So far, when I run any of the benchmarks or Shortest path examples, I
>>> always
>>> > get the LocalJobRunner error :
>>> >
>>> >
>>> >
>>> ---------------------------------------------------------------------------------------
>>> > Exception in thread "main" java.lang.IllegalArgumentException:
>>> > checkLocalJobRunnerConfiguration: When using LocalJobRunner, you
>>> cannot run
>>> > in split master / worker mode since there is only 1 task at a time!
>>> > at
>>> >
>>> org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:168)
>>> > at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:236)
>>> > at
>>> org.apache.giraph.benchmark.GiraphBenchmark.run(GiraphBenchmark.java:96)
>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>> > at
>>> >
>>> org.apache.giraph.benchmark.PageRankBenchmark.main(PageRankBenchmark.java:158)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > at java.lang.reflect.Method.invoke(Method.java:606)
>>> > at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>>> > at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>>> > -------------------------------------------------------------------
>>> >
>>> > I have tried some solutions from forums/StackOverflow/lists, but so far
>>> > nothing has worked. As far as I can tell, Hadoop is configured right
>>> (other
>>> > MR benchmarks run fine). I tried changing the above Giraph code file
>>> (making
>>> > the check function returned what I wanted), but that only starts the
>>> Job and
>>> > gets it running on a single machine - it never uses more than 1
>>> machine.
>>> >
>>> > Any help or pointers in the right direction would be much appreciated.
>>> >
>>> > Thanks,
>>> > Kenrick
>>>
>>
>>
>
Re: Giraph 1.1.0 not running on full cluster with Hadoop 2.6.0
Posted by Steven Harenberg <sd...@ncsu.edu>.
Hey Kenrick,
For the issue with GiraphApplicationMaster, I followed what Phillip did
here:
http://mail-archives.apache.org/mod_mbox/giraph-user/201503.mbox/%3CCAO3ErG_obGV8mELzX1j%2Be%3DaL6C%3D6%3DtdiSOVRBia2gh0H9tYLZA%40mail.gmail.com%3E
Basically you need the jar for giraph-examples to be in the directory where
you are issuing the command. You can do this by creating a symbolic link. I
have no idea why this worked and you can't use an absolute path, but that
is how it was for me.
Thanks,
Steve
On Mon, Mar 23, 2015 at 7:06 PM, Kenrick Fernandes <ke...@gmail.com>
wrote:
> Hi Phil,
>
> The build was successful - now running the *ShortestPaths* example gives
> me a different error,
> *"GiraphApplicationMaster* not found" . However, when I run the PageRank
> benchmark, I still
> get the same *LocalJobRunner* error:
>
> -------------------------
> *Command:*
> hadoop jar giraph-1.1.0-for-hadoop-2.6.0-jar-with-dependencies.jar
> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000 -w 30
>
> *Error:*
> Exception in thread "main" java.lang.IllegalArgumentException:
> checkLocalJobRunnerConfiguration: When using LocalJobRunner, must have only
> one worker since only 1 task at a time!
> at
> org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:162)
> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:236)
> at org.apache.giraph.benchmark.GiraphBenchmark.run(GiraphBenchmark.java:96)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.giraph.benchmark.PageRankBenchmark.main(PageRankBenchmark.java:158)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> -------------------------
>
> Did PageRank run fine for you ?
>
> Thanks,
> Kenrick
>
> On Mon, Mar 23, 2015 at 4:35 PM, Phillip Rhodes <motley.crue.fan@gmail.com
> > wrote:
>
>> What I had to do to get this to work was:
>>
>> edit the pom.xml and change the hadoop_yarn profile to remove the one
>> munge symbol that was something like _SASL_SOMETHING_OR_OTHER.
>>
>> Build using mvn -Phadoop_yarn -Dhadoop.version=2.5.2 (in my case)
>>
>>
>> Phil
>>
>> This message optimized for indexing by NSA PRISM
>>
>>
>> On Sun, Mar 22, 2015 at 4:28 PM, Kenrick Fernandes
>> <ke...@gmail.com> wrote:
>> > Hi,
>> >
>> > I am working with Giraph 1.1.0 and a YARN cluster with Hadoop 2.6.0.
>> > I build Giraph with
>> > mvn -Phadoop_2 -Dhadoop.version=2.6.0 clean package -DskipTests
>> >
>> > So far, when I run any of the benchmarks or Shortest path examples, I
>> always
>> > get the LocalJobRunner error :
>> >
>> >
>> >
>> ---------------------------------------------------------------------------------------
>> > Exception in thread "main" java.lang.IllegalArgumentException:
>> > checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot
>> run
>> > in split master / worker mode since there is only 1 task at a time!
>> > at
>> >
>> org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:168)
>> > at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:236)
>> > at
>> org.apache.giraph.benchmark.GiraphBenchmark.run(GiraphBenchmark.java:96)
>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> > at
>> >
>> org.apache.giraph.benchmark.PageRankBenchmark.main(PageRankBenchmark.java:158)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:606)
>> > at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>> > at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>> > -------------------------------------------------------------------
>> >
>> > I have tried some solutions from forums/StackOverflow/lists, but so far
>> > nothing has worked. As far as I can tell, Hadoop is configured right
>> (other
>> > MR benchmarks run fine). I tried changing the above Giraph code file
>> (making
>> > the check function returned what I wanted), but that only starts the
>> Job and
>> > gets it running on a single machine - it never uses more than 1 machine.
>> >
>> > Any help or pointers in the right direction would be much appreciated.
>> >
>> > Thanks,
>> > Kenrick
>>
>
>
Re: Giraph 1.1.0 not running on full cluster with Hadoop 2.6.0
Posted by Kenrick Fernandes <ke...@gmail.com>.
Hi Phil,
The build was successful - now running the *ShortestPaths* example gives me
a different error,
*"GiraphApplicationMaster* not found" . However, when I run the PageRank
benchmark, I still
get the same *LocalJobRunner* error:
-------------------------
*Command:*
hadoop jar giraph-1.1.0-for-hadoop-2.6.0-jar-with-dependencies.jar
org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000 -w 30
*Error:*
Exception in thread "main" java.lang.IllegalArgumentException:
checkLocalJobRunnerConfiguration: When using LocalJobRunner, must have only
one worker since only 1 task at a time!
at
org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:162)
at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:236)
at org.apache.giraph.benchmark.GiraphBenchmark.run(GiraphBenchmark.java:96)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.giraph.benchmark.PageRankBenchmark.main(PageRankBenchmark.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
-------------------------
Did PageRank run fine for you ?
Thanks,
Kenrick
On Mon, Mar 23, 2015 at 4:35 PM, Phillip Rhodes <mo...@gmail.com>
wrote:
> What I had to do to get this to work was:
>
> edit the pom.xml and change the hadoop_yarn profile to remove the one
> munge symbol that was something like _SASL_SOMETHING_OR_OTHER.
>
> Build using mvn -Phadoop_yarn -Dhadoop.version=2.5.2 (in my case)
>
>
> Phil
>
> This message optimized for indexing by NSA PRISM
>
>
> On Sun, Mar 22, 2015 at 4:28 PM, Kenrick Fernandes
> <ke...@gmail.com> wrote:
> > Hi,
> >
> > I am working with Giraph 1.1.0 and a YARN cluster with Hadoop 2.6.0.
> > I build Giraph with
> > mvn -Phadoop_2 -Dhadoop.version=2.6.0 clean package -DskipTests
> >
> > So far, when I run any of the benchmarks or Shortest path examples, I
> always
> > get the LocalJobRunner error :
> >
> >
> >
> ---------------------------------------------------------------------------------------
> > Exception in thread "main" java.lang.IllegalArgumentException:
> > checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot
> run
> > in split master / worker mode since there is only 1 task at a time!
> > at
> >
> org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:168)
> > at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:236)
> > at
> org.apache.giraph.benchmark.GiraphBenchmark.run(GiraphBenchmark.java:96)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> > at
> >
> org.apache.giraph.benchmark.PageRankBenchmark.main(PageRankBenchmark.java:158)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> > -------------------------------------------------------------------
> >
> > I have tried some solutions from forums/StackOverflow/lists, but so far
> > nothing has worked. As far as I can tell, Hadoop is configured right
> (other
> > MR benchmarks run fine). I tried changing the above Giraph code file
> (making
> > the check function returned what I wanted), but that only starts the Job
> and
> > gets it running on a single machine - it never uses more than 1 machine.
> >
> > Any help or pointers in the right direction would be much appreciated.
> >
> > Thanks,
> > Kenrick
>
Re: Giraph 1.1.0 not running on full cluster with Hadoop 2.6.0
Posted by Phillip Rhodes <mo...@gmail.com>.
What I had to do to get this to work was:
edit the pom.xml and change the hadoop_yarn profile to remove the one
munge symbol that was something like _SASL_SOMETHING_OR_OTHER.
Build using mvn -Phadoop_yarn -Dhadoop.version=2.5.2 (in my case)
Phil
This message optimized for indexing by NSA PRISM
On Sun, Mar 22, 2015 at 4:28 PM, Kenrick Fernandes
<ke...@gmail.com> wrote:
> Hi,
>
> I am working with Giraph 1.1.0 and a YARN cluster with Hadoop 2.6.0.
> I build Giraph with
> mvn -Phadoop_2 -Dhadoop.version=2.6.0 clean package -DskipTests
>
> So far, when I run any of the benchmarks or Shortest path examples, I always
> get the LocalJobRunner error :
>
>
> ---------------------------------------------------------------------------------------
> Exception in thread "main" java.lang.IllegalArgumentException:
> checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run
> in split master / worker mode since there is only 1 task at a time!
> at
> org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:168)
> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:236)
> at org.apache.giraph.benchmark.GiraphBenchmark.run(GiraphBenchmark.java:96)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.giraph.benchmark.PageRankBenchmark.main(PageRankBenchmark.java:158)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> -------------------------------------------------------------------
>
> I have tried some solutions from forums/StackOverflow/lists, but so far
> nothing has worked. As far as I can tell, Hadoop is configured right (other
> MR benchmarks run fine). I tried changing the above Giraph code file (making
> the check function returned what I wanted), but that only starts the Job and
> gets it running on a single machine - it never uses more than 1 machine.
>
> Any help or pointers in the right direction would be much appreciated.
>
> Thanks,
> Kenrick