You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Rohit Bhat <ro...@gmail.com> on 2013/12/03 01:30:01 UTC
Issue while executing SimpleShortestPathsComputation
Hi,
I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have installed
Giraph on top of it. I have been trying to
execute SimpleShortestPathsComputation example but i am getting the error:
"java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration: When
using LocalJobRunner, you cannot run in split master / worker mode since
there is only 1 task at a time!"
I fixed this by adding an entry in mapred-site.xml for
mapreduce.jobtracker.address but now i am getting a different error:
*Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
interface org.apache.hadoop.mapreduce.JobContext, but class was expected*
at
org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
at
org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
*Is Giraph compatible with yarn? If yes, am i missing anything?*
Thanks,
Rohit
Re: Issue while executing SimpleShortestPathsComputation
Posted by Rohit Bhat <ro...@gmail.com>.
It worked!! Thanks.
On Fri, Dec 20, 2013 at 4:56 PM, Alexandre Fonseca <
alexandrejorgefonseca@gmail.com> wrote:
> Hi,
>
> I submitted a patch for this some time ago at
> https://issues.apache.org/jira/browse/GIRAPH-811
>
> It should fix your problem, hopefully.
>
> Regards,
> Alexandre
> On 20 Dec 2013 23:42, "Rohit Bhat" <ro...@gmail.com> wrote:
>
>> Hi Alexandre,
>>
>> I had made the suggested change in mapred-site.xml and now the job
>> atleast seems to start but it does not complete.
>>
>> I am trying out the SimpleShortestPathsComputation example. It remains in
>> the running state and the following message is displayed on stdout:
>>
>> 13/12/20 14:39:32 INFO yarn.GiraphYarnClient: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 514.07
>> secs
>> 13/12/20 14:39:32 INFO yarn.GiraphYarnClient:
>> appattempt_1387571436273_0001_000001, State: RUNNING, Containers used: 2
>>
>> In the log file i see this message:
>>
>> 2013-12-20 14:31:10,891 INFO [org.apache.giraph.master.MasterThread]
>> master.BspServiceMaster (BspServiceMaster.java:cleanUpZooKeeper(1711)) -
>> cleanUpZooKeeper: Node
>> /_hadoopBsp/giraph_yarn_application_1387571436273_0001/_cleanedUpDir a$
>> 2013-12-20 14:31:10,900 INFO [org.apache.giraph.master.MasterThread]
>> master.BspServiceMaster (BspServiceMaster.java:cleanUpZooKeeper(1735)) -
>> cleanUpZooKeeper: Got 2 of 1 desired children from
>> /_hadoopBsp/giraph_yarn_application_138757$
>> $7571436273_0001/_cleanedUpDir to change since only got 2 nodes.
>>
>> Any reason why this is happening?
>>
>> Thanks,
>> Rohit
>>
>>
>> On Mon, Dec 9, 2013 at 3:45 PM, Alexandre Fonseca <
>> alexandrejorgefonseca@gmail.com> wrote:
>>
>>> I have confirmed a bug in Giraph's YARN code related to the default
>>> MapReduce
>>> configuration. Either provide your own mapreduce.application.classpath in
>>> mapred-site.xml or apply the patch described here:
>>> https://issues.apache.org/jira/browse/GIRAPH-814
>>>
>>> Application runs successfully if using custom
>>> mapreduce.application.classpath
>>> or no default mapreduce.application.classpath + patch. Application fails
>>> if
>>> using default mapreduce.application.classpath and no patch.
>>>
>>> Cheers,
>>> Alex
>>>
>>> On Monday 09 December 2013 22:05:20 Alexandre Fonseca wrote:
>>> > mapreduce.application.classpath
>>>
>>
>>
Re: Issue while executing SimpleShortestPathsComputation
Posted by Alexandre Fonseca <al...@gmail.com>.
Hi,
I submitted a patch for this some time ago at
https://issues.apache.org/jira/browse/GIRAPH-811
It should fix your problem, hopefully.
Regards,
Alexandre
On 20 Dec 2013 23:42, "Rohit Bhat" <ro...@gmail.com> wrote:
> Hi Alexandre,
>
> I had made the suggested change in mapred-site.xml and now the job atleast
> seems to start but it does not complete.
>
> I am trying out the SimpleShortestPathsComputation example. It remains in
> the running state and the following message is displayed on stdout:
>
> 13/12/20 14:39:32 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 514.07
> secs
> 13/12/20 14:39:32 INFO yarn.GiraphYarnClient:
> appattempt_1387571436273_0001_000001, State: RUNNING, Containers used: 2
>
> In the log file i see this message:
>
> 2013-12-20 14:31:10,891 INFO [org.apache.giraph.master.MasterThread]
> master.BspServiceMaster (BspServiceMaster.java:cleanUpZooKeeper(1711)) -
> cleanUpZooKeeper: Node
> /_hadoopBsp/giraph_yarn_application_1387571436273_0001/_cleanedUpDir a$
> 2013-12-20 14:31:10,900 INFO [org.apache.giraph.master.MasterThread]
> master.BspServiceMaster (BspServiceMaster.java:cleanUpZooKeeper(1735)) -
> cleanUpZooKeeper: Got 2 of 1 desired children from
> /_hadoopBsp/giraph_yarn_application_138757$
> $7571436273_0001/_cleanedUpDir to change since only got 2 nodes.
>
> Any reason why this is happening?
>
> Thanks,
> Rohit
>
>
> On Mon, Dec 9, 2013 at 3:45 PM, Alexandre Fonseca <
> alexandrejorgefonseca@gmail.com> wrote:
>
>> I have confirmed a bug in Giraph's YARN code related to the default
>> MapReduce
>> configuration. Either provide your own mapreduce.application.classpath in
>> mapred-site.xml or apply the patch described here:
>> https://issues.apache.org/jira/browse/GIRAPH-814
>>
>> Application runs successfully if using custom
>> mapreduce.application.classpath
>> or no default mapreduce.application.classpath + patch. Application fails
>> if
>> using default mapreduce.application.classpath and no patch.
>>
>> Cheers,
>> Alex
>>
>> On Monday 09 December 2013 22:05:20 Alexandre Fonseca wrote:
>> > mapreduce.application.classpath
>>
>
>
Re: Issue while executing SimpleShortestPathsComputation
Posted by Rohit Bhat <ro...@gmail.com>.
Hi Alexandre,
I had made the suggested change in mapred-site.xml and now the job atleast
seems to start but it does not complete.
I am trying out the SimpleShortestPathsComputation example. It remains in
the running state and the following message is displayed on stdout:
13/12/20 14:39:32 INFO yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 514.07
secs
13/12/20 14:39:32 INFO yarn.GiraphYarnClient:
appattempt_1387571436273_0001_000001, State: RUNNING, Containers used: 2
In the log file i see this message:
2013-12-20 14:31:10,891 INFO [org.apache.giraph.master.MasterThread]
master.BspServiceMaster (BspServiceMaster.java:cleanUpZooKeeper(1711)) -
cleanUpZooKeeper: Node
/_hadoopBsp/giraph_yarn_application_1387571436273_0001/_cleanedUpDir a$
2013-12-20 14:31:10,900 INFO [org.apache.giraph.master.MasterThread]
master.BspServiceMaster (BspServiceMaster.java:cleanUpZooKeeper(1735)) -
cleanUpZooKeeper: Got 2 of 1 desired children from
/_hadoopBsp/giraph_yarn_application_138757$
$7571436273_0001/_cleanedUpDir to change since only got 2 nodes.
Any reason why this is happening?
Thanks,
Rohit
On Mon, Dec 9, 2013 at 3:45 PM, Alexandre Fonseca <
alexandrejorgefonseca@gmail.com> wrote:
> I have confirmed a bug in Giraph's YARN code related to the default
> MapReduce
> configuration. Either provide your own mapreduce.application.classpath in
> mapred-site.xml or apply the patch described here:
> https://issues.apache.org/jira/browse/GIRAPH-814
>
> Application runs successfully if using custom
> mapreduce.application.classpath
> or no default mapreduce.application.classpath + patch. Application fails if
> using default mapreduce.application.classpath and no patch.
>
> Cheers,
> Alex
>
> On Monday 09 December 2013 22:05:20 Alexandre Fonseca wrote:
> > mapreduce.application.classpath
>
Re: Issue while executing SimpleShortestPathsComputation
Posted by Alexandre Fonseca <al...@gmail.com>.
I have confirmed a bug in Giraph's YARN code related to the default MapReduce
configuration. Either provide your own mapreduce.application.classpath in
mapred-site.xml or apply the patch described here:
https://issues.apache.org/jira/browse/GIRAPH-814
Application runs successfully if using custom mapreduce.application.classpath
or no default mapreduce.application.classpath + patch. Application fails if
using default mapreduce.application.classpath and no patch.
Cheers,
Alex
On Monday 09 December 2013 22:05:20 Alexandre Fonseca wrote:
> mapreduce.application.classpath
Re: Issue while executing SimpleShortestPathsComputation
Posted by Alexandre Fonseca <al...@gmail.com>.
GIRAPH-814 fixes the exception:
(GiraphApplicationMaster.java:main(442)) - GiraphApplicationMaster caught a t$
java.lang.NoClassDefFoundError:
org/apache/hadoop/mapreduce/lib/output/TextOutputFormat
which you'd supposedly get after you solved your current problem with the
GiraphApplicationMaster.
I see that you've already confirmed in a different mail that
GiraphApplicationMaster is indeed inside your giraph-example.jar so the only
thing that I can think of is that something might be wrong with your -yj
parameter (can you put the entire command you use here please?) or something
else is preventing Giraph from loading it.
Looking at the log that appears directly in your terminal when you run the
hadoop jar command, do you see something like the following?
13/12/10 10:56:01 INFO yarn.GiraphYarnClient: LIB JARS :giraph-examples-1.1.0-
SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
13/12/10 10:56:01 INFO yarn.YarnUtils: Class path name .
13/12/10 10:56:01 INFO yarn.YarnUtils: base path checking .
13/12/10 10:56:01 INFO yarn.GiraphYarnClient: Made local resource for
:/home/alex/Projects/giraph/example-runner/giraph-examples-1.1.0-SNAPSHOT-for-
hadoop-2.2.0-jar-with-dependencies.jar to
hdfs://localhost/user/alex/giraph_yarn_jar_cache/application_1386669300464_0001/giraph-
examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
13/12/10 10:56:02 INFO yarn.YarnUtils: Registered file in LocalResources ::
hdfs://localhost/user/alex/giraph_yarn_jar_cache/application_1386669300464_0001/giraph-
examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
On Tuesday 10 December 2013 09:39:34 Rob Vesse wrote:
> Applying the GIRAPH-814 patch did not help for me
>
> Even with the -yj argument (which fwiw I've been using all along) I still
> get GiraphApplicationMaster not found error
>
> Rob
>
> On 09/12/2013 21:45, "Alexandre Fonseca" <al...@gmail.com>
>
> wrote:
> >I have confirmed a bug in Giraph's YARN code related to the default
> >MapReduce
> >configuration. Either provide your own mapreduce.application.classpath in
> >mapred-site.xml or apply the patch described here:
> >https://issues.apache.org/jira/browse/GIRAPH-814
> >
> >Application runs successfully if using custom
> >mapreduce.application.classpath
> >or no default mapreduce.application.classpath + patch. Application fails
> >if
> >using default mapreduce.application.classpath and no patch.
> >
> >Cheers,
> >Alex
> >
> >On Monday 09 December 2013 22:05:20 Alexandre Fonseca wrote:
> >> mapreduce.application.classpath
Re: Issue while executing SimpleShortestPathsComputation
Posted by Rob Vesse <rv...@dotnetrdf.org>.
Applying the GIRAPH-814 patch did not help for me
Even with the -yj argument (which fwiw I've been using all along) I still
get GiraphApplicationMaster not found error
Rob
On 09/12/2013 21:45, "Alexandre Fonseca" <al...@gmail.com>
wrote:
>I have confirmed a bug in Giraph's YARN code related to the default
>MapReduce
>configuration. Either provide your own mapreduce.application.classpath in
>mapred-site.xml or apply the patch described here:
>https://issues.apache.org/jira/browse/GIRAPH-814
>
>Application runs successfully if using custom
>mapreduce.application.classpath
>or no default mapreduce.application.classpath + patch. Application fails
>if
>using default mapreduce.application.classpath and no patch.
>
>Cheers,
>Alex
>
>On Monday 09 December 2013 22:05:20 Alexandre Fonseca wrote:
>> mapreduce.application.classpath
>
Re: Issue while executing SimpleShortestPathsComputation
Posted by Alexandre Fonseca <al...@gmail.com>.
When you execute the job, one of the INFO lines should state the environment
used for the launching of the ApplicationMaster. In my case, it looks
something like this:
13/12/09 21:54:56 INFO yarn.GiraphYarnClient: Environment for AM :
{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:
$HADOOP_COMMON_HOME/share/hadoop/common/*:
$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:
$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:
$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:
$HADOOP_YARN_HOME/share/hadoop/yarn/*:
$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
So assuming that HADOOP_MAPRED_HOME is defined and pointing to the right
directory (HADOOP_HOME or HADOOP_PREFIX or HADOOP_COMMON_HOME) in the
node where the ApplicationMaster is launched, it should be able to pick up the
mapred-core jar.
Oh wait! I forgot I had to do something else. Can you confirm that in your case
the CLASSPATH actually shows the following in the last part?
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
Notice the comma instead of a colon.
I got that when I relied on the default settings. Adding the following to
mapred-site.xml turned the comma into a colon again:
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
I thought this was just a quirk with my configuration but if you can replicate
it's probably a MapReduce Configuration bug.
On Monday 09 December 2013 12:45:17 Vishal Mishra wrote:
> Tried the steps (one variation was 'clean package' instead of 'compile'
> since a dependency check was failing) provided by Alexandre and hitting the
> same error as Rohit.
> Note that I'm attempting on a cluster.
>
> Cmdline: hadoop jar
> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoo
> p-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip /user/hduser/rrdata/tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /user/hduser/rrdata/output -w 1 -yj
> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoo
> p-2.2.0-jar-with-dependencies.jar -ca giraph.SplitMasterWorker=false -ca
> giraph.zkList=exotica:2181
>
> On Mon, Dec 9, 2013 at 10:34 AM, Rohit Bhat <ro...@gmail.com>wrote:
> > Hi Alexandre,
> >
> > I looked into the logs and found this:
> >
> > 2013-12-09 12:29:08,925 INFO [main] yarn.GiraphApplicationMaster
> > (GiraphApplicationMaster.java:main(426)) - Starting GitaphAM
> > 2013-12-09 12:29:09,727 WARN [main] util.NativeCodeLoader
> > (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop
> > library
> > for your$
> > 2013-12-09 12:29:10,206 ERROR [main] yarn.GiraphApplicationMaster
> > (GiraphApplicationMaster.java:main(442)) - GiraphApplicationMaster caught
> > a
> > t$
> > java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/lib/output/
> > *TextOutputFormat*
> >
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:788)
> > at
> >
> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> >
> > In gam-stderr.log i found this:
> >
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> > [jar:file:/scratch/rohit/hadoop-2.2.0/tmp/nm-local-dir/usercache/rohit/app
> > cache/application_1386613677359_0001/filecach$ SLF4J: Found binding in
> > [jar:file:/scratch/rohit/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j1
> > 2-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinde$
> >
> > I tried the fix that you mentioned of linking and adding -yj for hadoop-
> > mapreduce-client-core-2.2.0.jar but that did not help. HADOOP_MAPRED_HOME
> > is pointing to HADOOP_HOME on my setup.
> >
> >
> > On Mon, Dec 9, 2013 at 10:11 AM, Alexandre Fonseca <
> >
> > alexandrejorgefonseca@gmail.com> wrote:
> >> Hi Rohit,
> >>
> >> In my case, I got 2 Shell$ExitCodeExceptions when trying to run the
> >> SimpleShortestPathsComputation:
> >>
> >> - Error: Could not find or load main class
> >> org.apache.giraph.yarn.GiraphApplicationMaster
> >>
> >> This was solved by adding -yj giraph-examples-1.1.0-SNAPSHOT-for-
> >>
> >> hadoop-2.2.0-jar-with-dependencies.jar
> >>
> >> - java.lang.NoClassDefFoundError:
> >> org/apache/hadoop/mapreduce/lib/output/TextOutputFormat
> >>
> >> This was solved by making sure HADOOP_MAPRED_HOME was
> >>
> >> pointing to the same directory as HADOOP_PREFIX or HADOOP_HOME and
> >> restarting
> >> both the nodemanager and resourcemanager. Another thing that also worked
> >> was
> >> linking the
> >> $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-client-
> >> core-2.2.0.jar file to the current directory and adding it to the -yj
> >>
> >> parameter:
> >> -yj giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-
> >>
> >> dependencies.jar,hadoop-mapreduce-client-core-2.2.0.jar
> >>
> >> I found the first error on:
> >>
> >> /home/alex/Programs/hadoop-2.2.0/logs/userlogs/application_1386443083254_
> >> 0001/container_ 1386443083254_0001_01_000001/gam-stderr.log
> >>
> >> And the second error on:
> >>
> >> /home/alex/Programs/hadoop-2.2.0/logs/userlogs/application_1386443083254_
> >> 0002/container_ 1386443083254_0001_01_000002/gam-stdout.log
> >>
> >> If you didn't change the default yarn property yarn.nodemanager.log-dirs,
> >> you
> >> should be able to see them on the respective directories on your side. If
> >> not,
> >> check on whatever you set that property to.
> >>
> >> Hope this helps!
> >>
> >> Alex
> >>
> >> On Monday 09 December 2013 09:50:35 Rohit Bhat wrote:
> >> > Hi Eli/Alexandre,
> >> >
> >> > I have tried with Giraph-1.1.0-SNAPSHOT and hadoop-2.2.0 as well using
> >>
> >> the
> >>
> >> > steps mentioned by Alexandre, but again i am getting the same error
> >>
> >> while
> >>
> >> > running the example:
> >> >
> >> > Command:
> >> >
> >> > $HADOOP_HOME/bin/hadoop jar
> >>
> >> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> >>
> >> > org.apache.giraph.GiraphRunner
> >> > org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> >> > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> >> > -vip /giraphInput/tiny_graph.txt -vof
> >> > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> >> > /output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca
> >> > giraph.zkList=localhost:2181 -yj
> >>
> >> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> >>
> >> > ERROR yarn.GiraphYarnClient: Giraph:
> >> > org.apache.giraph.examples.SimpleShortestPathsComputation reports
> >> > FAILED
> >> > state, diagnostics show: Application application_1386601013698_0005
> >>
> >> failed
> >>
> >> > 2 times due to AM Container for appattempt_1386601013698_0005_000002
> >>
> >> exited
> >>
> >> > with exitCode: 1 due to: Exception from container-launch:
> >> > org.apache.hadoop.util.Shell$ExitCodeException:
> >> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> >> > at org.apache.hadoop.util.Shell.run(Shell.java:379)
> >> > at
> >>
> >> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
> >>
> >> > at
> >>
> >> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launch
> >> Con>>
> >> > tainer(DefaultContainerExecutor.java:195) at
> >>
> >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Conta
> >> ine>>
> >> > rLaunch.call(ContainerLaunch.java:283) at
> >>
> >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Conta
> >> ine>>
> >> > rLaunch.call(ContainerLaunch.java:79) at
> >> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at
> >> > java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >> > at
> >>
> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
> >> :11>>
> >> > 45) at
> >>
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
> >> a:6>>
> >> > 15) at java.lang.Thread.run(Thread.java:724)
> >> >
> >> >
> >> > .Failing this attempt.. Failing the application.
> >> >
> >> > I checked the Hadoop logs as well but couldn't find anything meaningful
> >> > there too.
> >> >
> >> > Thanks,
> >> > Rohit
> >> >
> >> > On Sun, Dec 8, 2013 at 12:02 PM, Eli Reisman
> >>
> >> <ap...@gmail.com>wrote:
> >> > > Hey all,
> >> > >
> >> > > So. I think from more recent emails Rohit got this figured out, but,
> >> > > a
> >> > > couple points just so they out there again:
> >> > >
> >> > > Giraph 1.0.x can be built on a "pure YARN" profile with Hadoop
> >> > > 2.0.03-alpha or (with a patch thats still floating around on
> >>
> >> GIRAPH-687
> >>
> >> > > ticket) against Hadoop 2.0.4 and 2.0.5 alpha.
> >> > >
> >> > > Much better way to go is to build against Hadoop 2.2.x line using
> >>
> >> Giraph
> >>
> >> > > 1.1.x's YARN profile. This features a stable API, some basic support
> >>
> >> for
> >>
> >> > > security, and a concurrency fix that could bite you when launching
> >>
> >> many
> >>
> >> > > workers.
> >> > >
> >> > > Finally, you _can_ run Giraph on MRv2 on YARN in which case you need
> >>
> >> to
> >>
> >> > > build against a Hadoop 2.x Giraph Maven profile but _not_ the YARN
> >> > > profile.
> >> > > This runs on top of MapReduce just as Giraph on Hadoop 1.x or 0.20.x
> >>
> >> do,
> >>
> >> > > but riding on Mapper task instead of running the workers in YARN
> >> > > containers.
> >> > >
> >> > >
> >> > >
> >> > > On Sat, Dec 7, 2013 at 8:53 PM, Alexandre Fonseca <
> >> > >
> >> > > alexandrejorgefonseca@gmail.com> wrote:
> >> > >> Hello Yong,
> >> > >>
> >> > >> I was able to compile and run Giraph 1.1.0-SNAPSHOT on top of
> >> > >> hadoop-2.2.0.
> >> > >>
> >> > >> Here's what I did:
> >> > >> git clone https://git-wip-us.apache.org/repos/asf/giraph.git
> >> > >> cd giraph
> >> > >> git revert ac93c3b6c5bce5f22b293b29df91663ca7d7ce63
> >> > >> mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -DskipTests compile
> >> > >>
> >> > >> With that I got valid jars which allowed me to run the
> >> > >> SimpleShortestPaths
> >> > >>
> >> > >> example in my single-node setup Hadoop 2.2.0 + ZooKeeper 3.4.5:
> >> > >> mkdir test_example
> >> > >> ln -s
> >> > >> $GIRAPH_PREFIX/giraph-examples/target/giraph-examples-1.1.0-
> >> > >>
> >> > >> SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> >> > >>
> >> > >> $HADOOP_PREFIX/bin/hadoop jar
> >>
> >> giraph-examples-1.1.0-SNAPSHOT-for-
> >>
> >> > >> hadoop-2.2.0-jar-with-dependencies.jar
> >> > >> org.apache.giraph.GiraphRunner
> >> > >> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> >>
> >> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> >>
> >> > >> -vip
> >> > >> giraph/input/tiny_graph.txt -vof
> >> > >> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> >> > >> giraph/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false
> >>
> >> -ca
> >>
> >> > >> giraph.zkList=localhost:2181 -yj giraph-examples-1.1.0-SNAPSHOT-for-
> >> > >> hadoop-2.2.0-jar-with-dependencies.jar
> >> > >>
> >> > >> ###########
> >> > >> Notes:
> >> > >> * Wasn't able to do mvn package as it complained about not having
> >> > >> the
> >> > >> dependencies giraph-hive, giraph-hbase, etc... I suspect this is
> >>
> >> just a
> >>
> >> > >> matter
> >> > >> of changing the main pom.xml to include those modules with the
> >> > >> hadoop
> >> > >> 2.2.0
> >> > >> target or remove those dependencies from the giraph-dist if they no
> >> > >> longer
> >> > >> apply.
> >> > >> * You may get some memory limit exceeded problems for the
> >> > >> ApplicationMaster.
> >> > >> To fix, either increase the yarn.nodemanager.vmem-pmem-ratio
> >>
> >> parameter in
> >>
> >> > >> yarn-
> >> > >> site.xml or apply my patch to prevent allocation of heap size equal
> >>
> >> to
> >>
> >> > >> total
> >> > >> memory size, described at:
> >> > >> https://issues.apache.org/jira/browse/GIRAPH-812
> >> > >> * In my case, the application didn't exit even after outputting the
> >> > >> result and
> >> > >> having recognized that all nodes voted to terminate. I've opened an
> >>
> >> issue
> >>
> >> > >> in JIRA regarding this so if you suffer from the same, you can apply
> >>
> >> that
> >>
> >> > >> patch
> >> > >> too: https://issues.apache.org/jira/browse/GIRAPH-811
> >> > >>
> >> > >> ###########
> >> > >> Relevant JIRA issues:
> >> > >> * https://issues.apache.org/jira/browse/GIRAPH-798
> >> > >> * https://issues.apache.org/jira/browse/GIRAPH-804
> >> > >> * https://issues.apache.org/jira/browse/GIRAPH-811
> >> > >> * https://issues.apache.org/jira/browse/GIRAPH-812
> >> > >>
> >> > >> Best regards,
> >> > >> Alexandre
> >> > >>
> >> > >> On Friday 06 December 2013 00:32:32 Yong Guo wrote:
> >> > >> > Hi,
> >> > >> >
> >> > >> > I also got this error when tried to build Giraph on top of
> >>
> >> hadoop-2.2.0
> >>
> >> > >> > using similar commands. Can anyone help?
> >> > >> >
> >> > >> > Thanks,
> >> > >> > Yong
> >> > >> >
> >> > >> >
> >> > >> > 2013/12/6 Rohit Bhat <ro...@gmail.com>
> >> > >> >
> >> > >> > > I already have yarn (hadoop_2.2.0) on my machine. I tried the
> >> > >>
> >> > >> following 2
> >> > >>
> >> > >> > > commands and got similar errors in both the cases:
> >> > >> > >
> >> > >> > > 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package
> >> > >>
> >> > >> -DskipTests"
> >> > >>
> >> > >> > > 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean
> >>
> >> install
> >>
> >> > >> > > package"
> >> > >> > >
> >> > >> > >
> >> > >> > > [INFO] Checking dependency versions
> >> > >> > >
> >> > >> > > [ERROR] Found a problem with the direct dependency
> >> > >> > > commons-httpclient:commons-httpclient of the current project
> >> > >> > >
> >> > >> > > Expected version is 3.0.1
> >> > >> > >
> >> > >> > > Resolved version is 3.0.1
> >> > >>
> >> > >> > > Version 3.1 was expected by artifact:
> >> > >> org.apache.hadoop:hadoop-common
> >> > >>
> >> > >> > > [ERROR] Found a problem with the dependency
> >>
> >> commons-lang:commons-lang
> >>
> >> > >> > > Resolved version is 2.4
> >> > >> > >
> >> > >> > > Version 2.4 was expected by artifact:
> >> > >> > > commons-configuration:commons-configuration
> >> > >> > >
> >> > >> > > Version 2.5 was expected by artifact:
> >> > >> org.apache.hadoop:hadoop-common
> >> > >>
> >> > >> > > [ERROR] Found a problem with the dependency
> >> > >> > > commons-logging:commons-logging
> >> > >> > >
> >> > >> > > Resolved version is 1.0.4
> >> > >>
> >> > >> > > Version 1.1.1 was expected by artifact:
> >> > >> org.apache.hadoop:hadoop-common
> >> > >>
> >> > >> > > [ERROR] Found a problem with the dependency io.netty:netty
> >> > >> > >
> >> > >> > > Resolved version is 3.5.3.Final
> >> > >> > >
> >> > >> > > Version 3.6.2.Final was expected by artifacts: org.apache.
> >> > >> > >
> >> > >> > > hadoop:hadoop-mapreduce-client-common, org.apache.
> >> > >> > > hadoop:hadoop-mapreduce-client-core
> >> > >> > >
> >> > >> > > [ERROR] Found a problem with the dependency
> >> > >>
> >> > >> org.apache.zookeeper:zookeeper
> >> > >>
> >> > >> > > Resolved version is 3.3.3
> >> > >> > >
> >> > >> > > Version 3.4.5 was expected by artifacts:
> >> > >> > > org.apache.hadoop:hadoop-common,
> >> > >> > >
> >> > >> > > org.apache.hadoop:hadoop-mapreduce-client-common
> >> > >> > >
> >> > >> > >
> >> > >> > > Am i missing something?
> >> > >> > >
> >> > >> > > On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar
> >> > >>
> >> > >> <am...@gmail.com>wrote:
> >> > >> > >> Yes. Use this:
> >> > >> > >>
> >> > >> > >> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true
> >>
> >> package
> >>
> >> > >> > >> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang
> >> > >>
> >> > >> <ma...@gmail.com>wrote:
> >> > >> > >>> I get the same error when I compile Giraph against the default
> >> > >>
> >> > >> hadoop
> >> > >>
> >> > >> > >>> version (0.20.203.0), but my project that uses Giraph uses a
> >>
> >> more
> >>
> >> > >> > >>> recent hadoop version. Did you set the hadoop version via a
> >>
> >> maven
> >>
> >> > >> > >>> profile
> >> > >> > >>> when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for
> >> > >> > >>> hadoop
> >> > >>
> >> > >> 1.0)?
> >> > >>
> >> > >> > >>> Presumably, there's a way to compile Giraph for Yarn. There's
> >>
> >> even
> >>
> >> > >> > >>> a
> >> > >> > >>> Yarn profile in the pom.xml. I personally haven't had success
> >> > >>
> >> > >> building
> >> > >>
> >> > >> > >>> it
> >> > >> > >>> though.
> >> > >> > >>>
> >> > >> > >>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat
> >> > >>
> >> > >> <ro...@gmail.com>wrote:
> >> > >> > >>>> Hi,
> >> > >> > >>>>
> >> > >> > >>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and
> >>
> >> have
> >>
> >> > >> > >>>> installed Giraph on top of it. I have been trying to
> >> > >> > >>>> execute SimpleShortestPathsComputation example but i am
> >>
> >> getting
> >>
> >> > >> > >>>> the
> >> > >> > >>>> error:
> >> > >>
> >> > >> > >>>> "java.lang.IllegalArgumentException:
> >> > >> checkLocalJobRunnerConfiguration:
> >> > >> > >>>> When using LocalJobRunner, you cannot run in split master /
> >>
> >> worker
> >>
> >> > >> mode
> >> > >>
> >> > >> > >>>> since there is only 1 task at a time!"
> >> > >> > >>>>
> >> > >> > >>>> I fixed this by adding an entry in mapred-site.xml for
> >> > >> > >>>> mapreduce.jobtracker.address but now i am getting a different
> >> > >>
> >> > >> error:
> >> > >> > >>>> *Exception in thread "main"
> >> > >> > >>>> java.lang.IncompatibleClassChangeError:
> >> > >> > >>>> Found interface org.apache.hadoop.mapreduce.JobContext, but
> >>
> >> class
> >>
> >> > >> was
> >> > >>
> >> > >> > >>>> expected*
> >> > >> > >>>>
> >> > >> > >>>> at
> >>
> >> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.
> >>
> >> > >> > >>>> java:43) at
> >>
> >> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:4
> >>
> >> > >> > >>>> 56)
> >> > >> > >>>>
> >> > >> > >>>> at
> >>
> >> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter
> >>
> >> > >> > >>>> .java:342) at
> >> > >> > >>>> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
> >> > >> > >>>>
> >> > >> > >>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
> >> > >> > >>>>
> >> > >> > >>>> at java.security.AccessController.doPrivileged(Native Method)
> >> > >> > >>>>
> >> > >> > >>>> at javax.security.auth.Subject.doAs(Subject.java:415)
> >> > >> > >>>>
> >> > >> > >>>> at
> >>
> >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
> >>
> >> > >> > >>>> on.java:1491)>>>>
> >> > >> > >>>>
> >> > >> > >>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
> >> > >> > >>>>
> >> > >> > >>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
> >> > >> > >>>>
> >> > >> > >>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
> >> > >> > >>>>
> >> > >> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >> > >> > >>>>
> >> > >> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> >> > >> > >>>>
> >> > >> > >>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
> >> > >> > >>>>
> >> > >> > >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>
> >> Method)
> >>
> >> > >> > >>>> at
> >>
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
> >>
> >> > >> > >>>> va:57)>>>>
> >> > >> > >>>>
> >> > >> > >>>> at
> >>
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> >>
> >> > >> > >>>> rImpl.java:43) at
> >>
> >> java.lang.reflect.Method.invoke(Method.java:606)
> >>
> >> > >> > >>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> >> > >> > >>>>
> >> > >> > >>>> *Is Giraph compatible with yarn? If yes, am i missing
> >>
> >> anything?*
> >>
> >> > >> > >>>> Thanks,
> >> > >> > >>>> Rohit
Re: Issue while executing SimpleShortestPathsComputation
Posted by Vishal Mishra <mi...@gmail.com>.
Tried the steps (one variation was 'clean package' instead of 'compile'
since a dependency check was failing) provided by Alexandre and hitting the
same error as Rohit.
Note that I'm attempting on a cluster.
Cmdline: hadoop jar
$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /user/hduser/rrdata/tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/hduser/rrdata/output -w 1 -yj
$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
-ca giraph.SplitMasterWorker=false -ca giraph.zkList=exotica:2181
On Mon, Dec 9, 2013 at 10:34 AM, Rohit Bhat <ro...@gmail.com>wrote:
> Hi Alexandre,
>
> I looked into the logs and found this:
>
> 2013-12-09 12:29:08,925 INFO [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:main(426)) - Starting GitaphAM
> 2013-12-09 12:29:09,727 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library
> for your$
> 2013-12-09 12:29:10,206 ERROR [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:main(442)) - GiraphApplicationMaster caught a
> t$
> java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/lib/output/
> *TextOutputFormat*
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:788)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>
> In gam-stderr.log i found this:
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/scratch/rohit/hadoop-2.2.0/tmp/nm-local-dir/usercache/rohit/appcache/application_1386613677359_0001/filecach$
> SLF4J: Found binding in
> [jar:file:/scratch/rohit/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinde$
>
> I tried the fix that you mentioned of linking and adding -yj for hadoop-
> mapreduce-client-core-2.2.0.jar but that did not help. HADOOP_MAPRED_HOME
> is pointing to HADOOP_HOME on my setup.
>
>
> On Mon, Dec 9, 2013 at 10:11 AM, Alexandre Fonseca <
> alexandrejorgefonseca@gmail.com> wrote:
>
>> Hi Rohit,
>>
>> In my case, I got 2 Shell$ExitCodeExceptions when trying to run the
>> SimpleShortestPathsComputation:
>>
>> - Error: Could not find or load main class
>> org.apache.giraph.yarn.GiraphApplicationMaster
>>
>> This was solved by adding -yj giraph-examples-1.1.0-SNAPSHOT-for-
>> hadoop-2.2.0-jar-with-dependencies.jar
>>
>> - java.lang.NoClassDefFoundError:
>> org/apache/hadoop/mapreduce/lib/output/TextOutputFormat
>>
>> This was solved by making sure HADOOP_MAPRED_HOME was
>> pointing to the same directory as HADOOP_PREFIX or HADOOP_HOME and
>> restarting
>> both the nodemanager and resourcemanager. Another thing that also worked
>> was
>> linking the $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-client-
>> core-2.2.0.jar file to the current directory and adding it to the -yj
>> parameter:
>> -yj giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-
>> dependencies.jar,hadoop-mapreduce-client-core-2.2.0.jar
>>
>> I found the first error on:
>>
>> /home/alex/Programs/hadoop-2.2.0/logs/userlogs/application_1386443083254_0001/container_
>> 1386443083254_0001_01_000001/gam-stderr.log
>>
>> And the second error on:
>>
>> /home/alex/Programs/hadoop-2.2.0/logs/userlogs/application_1386443083254_0002/container_
>> 1386443083254_0001_01_000002/gam-stdout.log
>>
>> If you didn't change the default yarn property yarn.nodemanager.log-dirs,
>> you
>> should be able to see them on the respective directories on your side. If
>> not,
>> check on whatever you set that property to.
>>
>> Hope this helps!
>>
>> Alex
>>
>> On Monday 09 December 2013 09:50:35 Rohit Bhat wrote:
>> > Hi Eli/Alexandre,
>> >
>> > I have tried with Giraph-1.1.0-SNAPSHOT and hadoop-2.2.0 as well using
>> the
>> > steps mentioned by Alexandre, but again i am getting the same error
>> while
>> > running the example:
>> >
>> > Command:
>> >
>> > $HADOOP_HOME/bin/hadoop jar
>> >
>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> > org.apache.giraph.GiraphRunner
>> > org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>> > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> > -vip /giraphInput/tiny_graph.txt -vof
>> > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> > /output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca
>> > giraph.zkList=localhost:2181 -yj
>> >
>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> >
>> > ERROR yarn.GiraphYarnClient: Giraph:
>> > org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
>> > state, diagnostics show: Application application_1386601013698_0005
>> failed
>> > 2 times due to AM Container for appattempt_1386601013698_0005_000002
>> exited
>> > with exitCode: 1 due to: Exception from container-launch:
>> > org.apache.hadoop.util.Shell$ExitCodeException:
>> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>> > at org.apache.hadoop.util.Shell.run(Shell.java:379)
>> > at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>> > at
>> >
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchCon
>> > tainer(DefaultContainerExecutor.java:195) at
>> >
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Containe
>> > rLaunch.call(ContainerLaunch.java:283) at
>> >
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Containe
>> > rLaunch.call(ContainerLaunch.java:79) at
>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at
>> > java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
>> > 45) at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
>> > 15) at java.lang.Thread.run(Thread.java:724)
>> >
>> >
>> > .Failing this attempt.. Failing the application.
>> >
>> > I checked the Hadoop logs as well but couldn't find anything meaningful
>> > there too.
>> >
>> > Thanks,
>> > Rohit
>> >
>> > On Sun, Dec 8, 2013 at 12:02 PM, Eli Reisman
>> <ap...@gmail.com>wrote:
>> > > Hey all,
>> > >
>> > > So. I think from more recent emails Rohit got this figured out, but, a
>> > > couple points just so they out there again:
>> > >
>> > > Giraph 1.0.x can be built on a "pure YARN" profile with Hadoop
>> > > 2.0.03-alpha or (with a patch thats still floating around on
>> GIRAPH-687
>> > > ticket) against Hadoop 2.0.4 and 2.0.5 alpha.
>> > >
>> > > Much better way to go is to build against Hadoop 2.2.x line using
>> Giraph
>> > > 1.1.x's YARN profile. This features a stable API, some basic support
>> for
>> > > security, and a concurrency fix that could bite you when launching
>> many
>> > > workers.
>> > >
>> > > Finally, you _can_ run Giraph on MRv2 on YARN in which case you need
>> to
>> > > build against a Hadoop 2.x Giraph Maven profile but _not_ the YARN
>> > > profile.
>> > > This runs on top of MapReduce just as Giraph on Hadoop 1.x or 0.20.x
>> do,
>> > > but riding on Mapper task instead of running the workers in YARN
>> > > containers.
>> > >
>> > >
>> > >
>> > > On Sat, Dec 7, 2013 at 8:53 PM, Alexandre Fonseca <
>> > >
>> > > alexandrejorgefonseca@gmail.com> wrote:
>> > >> Hello Yong,
>> > >>
>> > >> I was able to compile and run Giraph 1.1.0-SNAPSHOT on top of
>> > >> hadoop-2.2.0.
>> > >>
>> > >> Here's what I did:
>> > >> git clone https://git-wip-us.apache.org/repos/asf/giraph.git
>> > >> cd giraph
>> > >> git revert ac93c3b6c5bce5f22b293b29df91663ca7d7ce63
>> > >> mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -DskipTests compile
>> > >>
>> > >> With that I got valid jars which allowed me to run the
>> > >> SimpleShortestPaths
>> > >>
>> > >> example in my single-node setup Hadoop 2.2.0 + ZooKeeper 3.4.5:
>> > >> mkdir test_example
>> > >> ln -s
>> > >> $GIRAPH_PREFIX/giraph-examples/target/giraph-examples-1.1.0-
>> > >>
>> > >> SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> > >>
>> > >> $HADOOP_PREFIX/bin/hadoop jar
>> giraph-examples-1.1.0-SNAPSHOT-for-
>> > >>
>> > >> hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner
>> > >> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>> > >>
>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> > >> -vip
>> > >> giraph/input/tiny_graph.txt -vof
>> > >> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> > >> giraph/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false
>> -ca
>> > >> giraph.zkList=localhost:2181 -yj giraph-examples-1.1.0-SNAPSHOT-for-
>> > >> hadoop-2.2.0-jar-with-dependencies.jar
>> > >>
>> > >> ###########
>> > >> Notes:
>> > >> * Wasn't able to do mvn package as it complained about not having the
>> > >> dependencies giraph-hive, giraph-hbase, etc... I suspect this is
>> just a
>> > >> matter
>> > >> of changing the main pom.xml to include those modules with the hadoop
>> > >> 2.2.0
>> > >> target or remove those dependencies from the giraph-dist if they no
>> > >> longer
>> > >> apply.
>> > >> * You may get some memory limit exceeded problems for the
>> > >> ApplicationMaster.
>> > >> To fix, either increase the yarn.nodemanager.vmem-pmem-ratio
>> parameter in
>> > >> yarn-
>> > >> site.xml or apply my patch to prevent allocation of heap size equal
>> to
>> > >> total
>> > >> memory size, described at:
>> > >> https://issues.apache.org/jira/browse/GIRAPH-812
>> > >> * In my case, the application didn't exit even after outputting the
>> > >> result and
>> > >> having recognized that all nodes voted to terminate. I've opened an
>> issue
>> > >> in JIRA regarding this so if you suffer from the same, you can apply
>> that
>> > >> patch
>> > >> too: https://issues.apache.org/jira/browse/GIRAPH-811
>> > >>
>> > >> ###########
>> > >> Relevant JIRA issues:
>> > >> * https://issues.apache.org/jira/browse/GIRAPH-798
>> > >> * https://issues.apache.org/jira/browse/GIRAPH-804
>> > >> * https://issues.apache.org/jira/browse/GIRAPH-811
>> > >> * https://issues.apache.org/jira/browse/GIRAPH-812
>> > >>
>> > >> Best regards,
>> > >> Alexandre
>> > >>
>> > >> On Friday 06 December 2013 00:32:32 Yong Guo wrote:
>> > >> > Hi,
>> > >> >
>> > >> > I also got this error when tried to build Giraph on top of
>> hadoop-2.2.0
>> > >> > using similar commands. Can anyone help?
>> > >> >
>> > >> > Thanks,
>> > >> > Yong
>> > >> >
>> > >> >
>> > >> > 2013/12/6 Rohit Bhat <ro...@gmail.com>
>> > >> >
>> > >> > > I already have yarn (hadoop_2.2.0) on my machine. I tried the
>> > >>
>> > >> following 2
>> > >>
>> > >> > > commands and got similar errors in both the cases:
>> > >> > >
>> > >> > > 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package
>> > >>
>> > >> -DskipTests"
>> > >>
>> > >> > > 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean
>> install
>> > >> > > package"
>> > >> > >
>> > >> > >
>> > >> > > [INFO] Checking dependency versions
>> > >> > >
>> > >> > > [ERROR] Found a problem with the direct dependency
>> > >> > > commons-httpclient:commons-httpclient of the current project
>> > >> > >
>> > >> > > Expected version is 3.0.1
>> > >> > >
>> > >> > > Resolved version is 3.0.1
>> > >>
>> > >> > > Version 3.1 was expected by artifact:
>> > >> org.apache.hadoop:hadoop-common
>> > >>
>> > >> > > [ERROR] Found a problem with the dependency
>> commons-lang:commons-lang
>> > >> > >
>> > >> > > Resolved version is 2.4
>> > >> > >
>> > >> > > Version 2.4 was expected by artifact:
>> > >> > > commons-configuration:commons-configuration
>> > >> > >
>> > >> > > Version 2.5 was expected by artifact:
>> > >> org.apache.hadoop:hadoop-common
>> > >>
>> > >> > > [ERROR] Found a problem with the dependency
>> > >> > > commons-logging:commons-logging
>> > >> > >
>> > >> > > Resolved version is 1.0.4
>> > >>
>> > >> > > Version 1.1.1 was expected by artifact:
>> > >> org.apache.hadoop:hadoop-common
>> > >>
>> > >> > > [ERROR] Found a problem with the dependency io.netty:netty
>> > >> > >
>> > >> > > Resolved version is 3.5.3.Final
>> > >> > >
>> > >> > > Version 3.6.2.Final was expected by artifacts: org.apache.
>> > >> > >
>> > >> > > hadoop:hadoop-mapreduce-client-common, org.apache.
>> > >> > > hadoop:hadoop-mapreduce-client-core
>> > >> > >
>> > >> > > [ERROR] Found a problem with the dependency
>> > >>
>> > >> org.apache.zookeeper:zookeeper
>> > >>
>> > >> > > Resolved version is 3.3.3
>> > >> > >
>> > >> > > Version 3.4.5 was expected by artifacts:
>> > >> > > org.apache.hadoop:hadoop-common,
>> > >> > >
>> > >> > > org.apache.hadoop:hadoop-mapreduce-client-common
>> > >> > >
>> > >> > >
>> > >> > > Am i missing something?
>> > >> > >
>> > >> > > On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar
>> > >>
>> > >> <am...@gmail.com>wrote:
>> > >> > >> Yes. Use this:
>> > >> > >>
>> > >> > >> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true
>> package
>> > >> > >>
>> > >> > >> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang
>> > >>
>> > >> <ma...@gmail.com>wrote:
>> > >> > >>> I get the same error when I compile Giraph against the default
>> > >>
>> > >> hadoop
>> > >>
>> > >> > >>> version (0.20.203.0), but my project that uses Giraph uses a
>> more
>> > >> > >>> recent hadoop version. Did you set the hadoop version via a
>> maven
>> > >> > >>> profile
>> > >> > >>> when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop
>> > >>
>> > >> 1.0)?
>> > >>
>> > >> > >>> Presumably, there's a way to compile Giraph for Yarn. There's
>> even
>> > >> > >>> a
>> > >> > >>> Yarn profile in the pom.xml. I personally haven't had success
>> > >>
>> > >> building
>> > >>
>> > >> > >>> it
>> > >> > >>> though.
>> > >> > >>>
>> > >> > >>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat
>> > >>
>> > >> <ro...@gmail.com>wrote:
>> > >> > >>>> Hi,
>> > >> > >>>>
>> > >> > >>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and
>> have
>> > >> > >>>> installed Giraph on top of it. I have been trying to
>> > >> > >>>> execute SimpleShortestPathsComputation example but i am
>> getting
>> > >> > >>>> the
>> > >> > >>>> error:
>> > >>
>> > >> > >>>> "java.lang.IllegalArgumentException:
>> > >> checkLocalJobRunnerConfiguration:
>> > >> > >>>> When using LocalJobRunner, you cannot run in split master /
>> worker
>> > >>
>> > >> mode
>> > >>
>> > >> > >>>> since there is only 1 task at a time!"
>> > >> > >>>>
>> > >> > >>>> I fixed this by adding an entry in mapred-site.xml for
>> > >> > >>>> mapreduce.jobtracker.address but now i am getting a different
>> > >>
>> > >> error:
>> > >> > >>>> *Exception in thread "main"
>> > >> > >>>> java.lang.IncompatibleClassChangeError:
>> > >> > >>>> Found interface org.apache.hadoop.mapreduce.JobContext, but
>> class
>> > >>
>> > >> was
>> > >>
>> > >> > >>>> expected*
>> > >> > >>>>
>> > >> > >>>> at
>> > >>
>> > >>
>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.
>> > >>
>> > >> > >>>> java:43) at
>> > >>
>> > >>
>> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:4
>> > >>
>> > >> > >>>> 56)
>> > >> > >>>>
>> > >> > >>>> at
>> > >>
>> > >>
>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter
>> > >>
>> > >> > >>>> .java:342) at
>> > >> > >>>> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>> > >> > >>>>
>> > >> > >>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>> > >> > >>>>
>> > >> > >>>> at java.security.AccessController.doPrivileged(Native Method)
>> > >> > >>>>
>> > >> > >>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> > >> > >>>>
>> > >> > >>>> at
>> > >>
>> > >>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
>> > >>
>> > >> > >>>> on.java:1491)>>>>
>> > >> > >>>>
>> > >> > >>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>> > >> > >>>>
>> > >> > >>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
>> > >> > >>>>
>> > >> > >>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>> > >> > >>>>
>> > >> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> > >> > >>>>
>> > >> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> > >> > >>>>
>> > >> > >>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>> > >> > >>>>
>> > >> > >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> > >> > >>>>
>> > >> > >>>> at
>> > >>
>> > >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
>> > >>
>> > >> > >>>> va:57)>>>>
>> > >> > >>>>
>> > >> > >>>> at
>> > >>
>> > >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
>> > >>
>> > >> > >>>> rImpl.java:43) at
>> java.lang.reflect.Method.invoke(Method.java:606)
>> > >> > >>>>
>> > >> > >>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>> > >> > >>>>
>> > >> > >>>> *Is Giraph compatible with yarn? If yes, am i missing
>> anything?*
>> > >> > >>>>
>> > >> > >>>> Thanks,
>> > >> > >>>> Rohit
>>
>
>
Re: Issue while executing SimpleShortestPathsComputation
Posted by Rohit Bhat <ro...@gmail.com>.
Hi Alexandre,
I looked into the logs and found this:
2013-12-09 12:29:08,925 INFO [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:main(426)) - Starting GitaphAM
2013-12-09 12:29:09,727 WARN [main] util.NativeCodeLoader
(NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library
for your$
2013-12-09 12:29:10,206 ERROR [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:main(442)) - GiraphApplicationMaster caught a
t$
java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/lib/output/
*TextOutputFormat*
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:788)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
In gam-stderr.log i found this:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/scratch/rohit/hadoop-2.2.0/tmp/nm-local-dir/usercache/rohit/appcache/application_1386613677359_0001/filecach$
SLF4J: Found binding in
[jar:file:/scratch/rohit/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinde$
I tried the fix that you mentioned of linking and adding -yj for hadoop-
mapreduce-client-core-2.2.0.jar but that did not help. HADOOP_MAPRED_HOME
is pointing to HADOOP_HOME on my setup.
On Mon, Dec 9, 2013 at 10:11 AM, Alexandre Fonseca <
alexandrejorgefonseca@gmail.com> wrote:
> Hi Rohit,
>
> In my case, I got 2 Shell$ExitCodeExceptions when trying to run the
> SimpleShortestPathsComputation:
>
> - Error: Could not find or load main class
> org.apache.giraph.yarn.GiraphApplicationMaster
>
> This was solved by adding -yj giraph-examples-1.1.0-SNAPSHOT-for-
> hadoop-2.2.0-jar-with-dependencies.jar
>
> - java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/lib/output/TextOutputFormat
>
> This was solved by making sure HADOOP_MAPRED_HOME was
> pointing to the same directory as HADOOP_PREFIX or HADOOP_HOME and
> restarting
> both the nodemanager and resourcemanager. Another thing that also worked
> was
> linking the $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-client-
> core-2.2.0.jar file to the current directory and adding it to the -yj
> parameter:
> -yj giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-
> dependencies.jar,hadoop-mapreduce-client-core-2.2.0.jar
>
> I found the first error on:
>
> /home/alex/Programs/hadoop-2.2.0/logs/userlogs/application_1386443083254_0001/container_
> 1386443083254_0001_01_000001/gam-stderr.log
>
> And the second error on:
>
> /home/alex/Programs/hadoop-2.2.0/logs/userlogs/application_1386443083254_0002/container_
> 1386443083254_0001_01_000002/gam-stdout.log
>
> If you didn't change the default yarn property yarn.nodemanager.log-dirs,
> you
> should be able to see them on the respective directories on your side. If
> not,
> check on whatever you set that property to.
>
> Hope this helps!
>
> Alex
>
> On Monday 09 December 2013 09:50:35 Rohit Bhat wrote:
> > Hi Eli/Alexandre,
> >
> > I have tried with Giraph-1.1.0-SNAPSHOT and hadoop-2.2.0 as well using
> the
> > steps mentioned by Alexandre, but again i am getting the same error while
> > running the example:
> >
> > Command:
> >
> > $HADOOP_HOME/bin/hadoop jar
> > giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> > org.apache.giraph.GiraphRunner
> > org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> > -vip /giraphInput/tiny_graph.txt -vof
> > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> > /output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca
> > giraph.zkList=localhost:2181 -yj
> > giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> >
> > ERROR yarn.GiraphYarnClient: Giraph:
> > org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
> > state, diagnostics show: Application application_1386601013698_0005
> failed
> > 2 times due to AM Container for appattempt_1386601013698_0005_000002
> exited
> > with exitCode: 1 due to: Exception from container-launch:
> > org.apache.hadoop.util.Shell$ExitCodeException:
> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> > at org.apache.hadoop.util.Shell.run(Shell.java:379)
> > at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
> > at
> >
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchCon
> > tainer(DefaultContainerExecutor.java:195) at
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Containe
> > rLaunch.call(ContainerLaunch.java:283) at
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Containe
> > rLaunch.call(ContainerLaunch.java:79) at
> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at
> > java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
> > 45) at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
> > 15) at java.lang.Thread.run(Thread.java:724)
> >
> >
> > .Failing this attempt.. Failing the application.
> >
> > I checked the Hadoop logs as well but couldn't find anything meaningful
> > there too.
> >
> > Thanks,
> > Rohit
> >
> > On Sun, Dec 8, 2013 at 12:02 PM, Eli Reisman
> <ap...@gmail.com>wrote:
> > > Hey all,
> > >
> > > So. I think from more recent emails Rohit got this figured out, but, a
> > > couple points just so they out there again:
> > >
> > > Giraph 1.0.x can be built on a "pure YARN" profile with Hadoop
> > > 2.0.03-alpha or (with a patch thats still floating around on GIRAPH-687
> > > ticket) against Hadoop 2.0.4 and 2.0.5 alpha.
> > >
> > > Much better way to go is to build against Hadoop 2.2.x line using
> Giraph
> > > 1.1.x's YARN profile. This features a stable API, some basic support
> for
> > > security, and a concurrency fix that could bite you when launching many
> > > workers.
> > >
> > > Finally, you _can_ run Giraph on MRv2 on YARN in which case you need to
> > > build against a Hadoop 2.x Giraph Maven profile but _not_ the YARN
> > > profile.
> > > This runs on top of MapReduce just as Giraph on Hadoop 1.x or 0.20.x
> do,
> > > but riding on Mapper task instead of running the workers in YARN
> > > containers.
> > >
> > >
> > >
> > > On Sat, Dec 7, 2013 at 8:53 PM, Alexandre Fonseca <
> > >
> > > alexandrejorgefonseca@gmail.com> wrote:
> > >> Hello Yong,
> > >>
> > >> I was able to compile and run Giraph 1.1.0-SNAPSHOT on top of
> > >> hadoop-2.2.0.
> > >>
> > >> Here's what I did:
> > >> git clone https://git-wip-us.apache.org/repos/asf/giraph.git
> > >> cd giraph
> > >> git revert ac93c3b6c5bce5f22b293b29df91663ca7d7ce63
> > >> mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -DskipTests compile
> > >>
> > >> With that I got valid jars which allowed me to run the
> > >> SimpleShortestPaths
> > >>
> > >> example in my single-node setup Hadoop 2.2.0 + ZooKeeper 3.4.5:
> > >> mkdir test_example
> > >> ln -s
> > >> $GIRAPH_PREFIX/giraph-examples/target/giraph-examples-1.1.0-
> > >>
> > >> SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> > >>
> > >> $HADOOP_PREFIX/bin/hadoop jar
> giraph-examples-1.1.0-SNAPSHOT-for-
> > >>
> > >> hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner
> > >> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> > >>
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> > >> -vip
> > >> giraph/input/tiny_graph.txt -vof
> > >> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> > >> giraph/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false
> -ca
> > >> giraph.zkList=localhost:2181 -yj giraph-examples-1.1.0-SNAPSHOT-for-
> > >> hadoop-2.2.0-jar-with-dependencies.jar
> > >>
> > >> ###########
> > >> Notes:
> > >> * Wasn't able to do mvn package as it complained about not having the
> > >> dependencies giraph-hive, giraph-hbase, etc... I suspect this is just
> a
> > >> matter
> > >> of changing the main pom.xml to include those modules with the hadoop
> > >> 2.2.0
> > >> target or remove those dependencies from the giraph-dist if they no
> > >> longer
> > >> apply.
> > >> * You may get some memory limit exceeded problems for the
> > >> ApplicationMaster.
> > >> To fix, either increase the yarn.nodemanager.vmem-pmem-ratio
> parameter in
> > >> yarn-
> > >> site.xml or apply my patch to prevent allocation of heap size equal to
> > >> total
> > >> memory size, described at:
> > >> https://issues.apache.org/jira/browse/GIRAPH-812
> > >> * In my case, the application didn't exit even after outputting the
> > >> result and
> > >> having recognized that all nodes voted to terminate. I've opened an
> issue
> > >> in JIRA regarding this so if you suffer from the same, you can apply
> that
> > >> patch
> > >> too: https://issues.apache.org/jira/browse/GIRAPH-811
> > >>
> > >> ###########
> > >> Relevant JIRA issues:
> > >> * https://issues.apache.org/jira/browse/GIRAPH-798
> > >> * https://issues.apache.org/jira/browse/GIRAPH-804
> > >> * https://issues.apache.org/jira/browse/GIRAPH-811
> > >> * https://issues.apache.org/jira/browse/GIRAPH-812
> > >>
> > >> Best regards,
> > >> Alexandre
> > >>
> > >> On Friday 06 December 2013 00:32:32 Yong Guo wrote:
> > >> > Hi,
> > >> >
> > >> > I also got this error when tried to build Giraph on top of
> hadoop-2.2.0
> > >> > using similar commands. Can anyone help?
> > >> >
> > >> > Thanks,
> > >> > Yong
> > >> >
> > >> >
> > >> > 2013/12/6 Rohit Bhat <ro...@gmail.com>
> > >> >
> > >> > > I already have yarn (hadoop_2.2.0) on my machine. I tried the
> > >>
> > >> following 2
> > >>
> > >> > > commands and got similar errors in both the cases:
> > >> > >
> > >> > > 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package
> > >>
> > >> -DskipTests"
> > >>
> > >> > > 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean
> install
> > >> > > package"
> > >> > >
> > >> > >
> > >> > > [INFO] Checking dependency versions
> > >> > >
> > >> > > [ERROR] Found a problem with the direct dependency
> > >> > > commons-httpclient:commons-httpclient of the current project
> > >> > >
> > >> > > Expected version is 3.0.1
> > >> > >
> > >> > > Resolved version is 3.0.1
> > >>
> > >> > > Version 3.1 was expected by artifact:
> > >> org.apache.hadoop:hadoop-common
> > >>
> > >> > > [ERROR] Found a problem with the dependency
> commons-lang:commons-lang
> > >> > >
> > >> > > Resolved version is 2.4
> > >> > >
> > >> > > Version 2.4 was expected by artifact:
> > >> > > commons-configuration:commons-configuration
> > >> > >
> > >> > > Version 2.5 was expected by artifact:
> > >> org.apache.hadoop:hadoop-common
> > >>
> > >> > > [ERROR] Found a problem with the dependency
> > >> > > commons-logging:commons-logging
> > >> > >
> > >> > > Resolved version is 1.0.4
> > >>
> > >> > > Version 1.1.1 was expected by artifact:
> > >> org.apache.hadoop:hadoop-common
> > >>
> > >> > > [ERROR] Found a problem with the dependency io.netty:netty
> > >> > >
> > >> > > Resolved version is 3.5.3.Final
> > >> > >
> > >> > > Version 3.6.2.Final was expected by artifacts: org.apache.
> > >> > >
> > >> > > hadoop:hadoop-mapreduce-client-common, org.apache.
> > >> > > hadoop:hadoop-mapreduce-client-core
> > >> > >
> > >> > > [ERROR] Found a problem with the dependency
> > >>
> > >> org.apache.zookeeper:zookeeper
> > >>
> > >> > > Resolved version is 3.3.3
> > >> > >
> > >> > > Version 3.4.5 was expected by artifacts:
> > >> > > org.apache.hadoop:hadoop-common,
> > >> > >
> > >> > > org.apache.hadoop:hadoop-mapreduce-client-common
> > >> > >
> > >> > >
> > >> > > Am i missing something?
> > >> > >
> > >> > > On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar
> > >>
> > >> <am...@gmail.com>wrote:
> > >> > >> Yes. Use this:
> > >> > >>
> > >> > >> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true
> package
> > >> > >>
> > >> > >> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang
> > >>
> > >> <ma...@gmail.com>wrote:
> > >> > >>> I get the same error when I compile Giraph against the default
> > >>
> > >> hadoop
> > >>
> > >> > >>> version (0.20.203.0), but my project that uses Giraph uses a
> more
> > >> > >>> recent hadoop version. Did you set the hadoop version via a
> maven
> > >> > >>> profile
> > >> > >>> when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop
> > >>
> > >> 1.0)?
> > >>
> > >> > >>> Presumably, there's a way to compile Giraph for Yarn. There's
> even
> > >> > >>> a
> > >> > >>> Yarn profile in the pom.xml. I personally haven't had success
> > >>
> > >> building
> > >>
> > >> > >>> it
> > >> > >>> though.
> > >> > >>>
> > >> > >>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat
> > >>
> > >> <ro...@gmail.com>wrote:
> > >> > >>>> Hi,
> > >> > >>>>
> > >> > >>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and
> have
> > >> > >>>> installed Giraph on top of it. I have been trying to
> > >> > >>>> execute SimpleShortestPathsComputation example but i am getting
> > >> > >>>> the
> > >> > >>>> error:
> > >>
> > >> > >>>> "java.lang.IllegalArgumentException:
> > >> checkLocalJobRunnerConfiguration:
> > >> > >>>> When using LocalJobRunner, you cannot run in split master /
> worker
> > >>
> > >> mode
> > >>
> > >> > >>>> since there is only 1 task at a time!"
> > >> > >>>>
> > >> > >>>> I fixed this by adding an entry in mapred-site.xml for
> > >> > >>>> mapreduce.jobtracker.address but now i am getting a different
> > >>
> > >> error:
> > >> > >>>> *Exception in thread "main"
> > >> > >>>> java.lang.IncompatibleClassChangeError:
> > >> > >>>> Found interface org.apache.hadoop.mapreduce.JobContext, but
> class
> > >>
> > >> was
> > >>
> > >> > >>>> expected*
> > >> > >>>>
> > >> > >>>> at
> > >>
> > >>
> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.
> > >>
> > >> > >>>> java:43) at
> > >>
> > >>
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:4
> > >>
> > >> > >>>> 56)
> > >> > >>>>
> > >> > >>>> at
> > >>
> > >>
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter
> > >>
> > >> > >>>> .java:342) at
> > >> > >>>> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
> > >> > >>>>
> > >> > >>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
> > >> > >>>>
> > >> > >>>> at java.security.AccessController.doPrivileged(Native Method)
> > >> > >>>>
> > >> > >>>> at javax.security.auth.Subject.doAs(Subject.java:415)
> > >> > >>>>
> > >> > >>>> at
> > >>
> > >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
> > >>
> > >> > >>>> on.java:1491)>>>>
> > >> > >>>>
> > >> > >>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
> > >> > >>>>
> > >> > >>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
> > >> > >>>>
> > >> > >>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
> > >> > >>>>
> > >> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > >> > >>>>
> > >> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> > >> > >>>>
> > >> > >>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
> > >> > >>>>
> > >> > >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >> > >>>>
> > >> > >>>> at
> > >>
> > >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
> > >>
> > >> > >>>> va:57)>>>>
> > >> > >>>>
> > >> > >>>> at
> > >>
> > >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> > >>
> > >> > >>>> rImpl.java:43) at
> java.lang.reflect.Method.invoke(Method.java:606)
> > >> > >>>>
> > >> > >>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> > >> > >>>>
> > >> > >>>> *Is Giraph compatible with yarn? If yes, am i missing
> anything?*
> > >> > >>>>
> > >> > >>>> Thanks,
> > >> > >>>> Rohit
>
Re: Issue while executing SimpleShortestPathsComputation
Posted by Alexandre Fonseca <al...@gmail.com>.
Hi Rohit,
In my case, I got 2 Shell$ExitCodeExceptions when trying to run the
SimpleShortestPathsComputation:
- Error: Could not find or load main class
org.apache.giraph.yarn.GiraphApplicationMaster
This was solved by adding -yj giraph-examples-1.1.0-SNAPSHOT-for-
hadoop-2.2.0-jar-with-dependencies.jar
- java.lang.NoClassDefFoundError:
org/apache/hadoop/mapreduce/lib/output/TextOutputFormat
This was solved by making sure HADOOP_MAPRED_HOME was
pointing to the same directory as HADOOP_PREFIX or HADOOP_HOME and restarting
both the nodemanager and resourcemanager. Another thing that also worked was
linking the $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-client-
core-2.2.0.jar file to the current directory and adding it to the -yj
parameter:
-yj giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-
dependencies.jar,hadoop-mapreduce-client-core-2.2.0.jar
I found the first error on:
/home/alex/Programs/hadoop-2.2.0/logs/userlogs/application_1386443083254_0001/container_
1386443083254_0001_01_000001/gam-stderr.log
And the second error on:
/home/alex/Programs/hadoop-2.2.0/logs/userlogs/application_1386443083254_0002/container_
1386443083254_0001_01_000002/gam-stdout.log
If you didn't change the default yarn property yarn.nodemanager.log-dirs, you
should be able to see them on the respective directories on your side. If not,
check on whatever you set that property to.
Hope this helps!
Alex
On Monday 09 December 2013 09:50:35 Rohit Bhat wrote:
> Hi Eli/Alexandre,
>
> I have tried with Giraph-1.1.0-SNAPSHOT and hadoop-2.2.0 as well using the
> steps mentioned by Alexandre, but again i am getting the same error while
> running the example:
>
> Command:
>
> $HADOOP_HOME/bin/hadoop jar
> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip /giraphInput/tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca
> giraph.zkList=localhost:2181 -yj
> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>
> ERROR yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
> state, diagnostics show: Application application_1386601013698_0005 failed
> 2 times due to AM Container for appattempt_1386601013698_0005_000002 exited
> with exitCode: 1 due to: Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchCon
> tainer(DefaultContainerExecutor.java:195) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Containe
> rLaunch.call(ContainerLaunch.java:283) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Containe
> rLaunch.call(ContainerLaunch.java:79) at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at
> java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
> 45) at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
> 15) at java.lang.Thread.run(Thread.java:724)
>
>
> .Failing this attempt.. Failing the application.
>
> I checked the Hadoop logs as well but couldn't find anything meaningful
> there too.
>
> Thanks,
> Rohit
>
> On Sun, Dec 8, 2013 at 12:02 PM, Eli Reisman
<ap...@gmail.com>wrote:
> > Hey all,
> >
> > So. I think from more recent emails Rohit got this figured out, but, a
> > couple points just so they out there again:
> >
> > Giraph 1.0.x can be built on a "pure YARN" profile with Hadoop
> > 2.0.03-alpha or (with a patch thats still floating around on GIRAPH-687
> > ticket) against Hadoop 2.0.4 and 2.0.5 alpha.
> >
> > Much better way to go is to build against Hadoop 2.2.x line using Giraph
> > 1.1.x's YARN profile. This features a stable API, some basic support for
> > security, and a concurrency fix that could bite you when launching many
> > workers.
> >
> > Finally, you _can_ run Giraph on MRv2 on YARN in which case you need to
> > build against a Hadoop 2.x Giraph Maven profile but _not_ the YARN
> > profile.
> > This runs on top of MapReduce just as Giraph on Hadoop 1.x or 0.20.x do,
> > but riding on Mapper task instead of running the workers in YARN
> > containers.
> >
> >
> >
> > On Sat, Dec 7, 2013 at 8:53 PM, Alexandre Fonseca <
> >
> > alexandrejorgefonseca@gmail.com> wrote:
> >> Hello Yong,
> >>
> >> I was able to compile and run Giraph 1.1.0-SNAPSHOT on top of
> >> hadoop-2.2.0.
> >>
> >> Here's what I did:
> >> git clone https://git-wip-us.apache.org/repos/asf/giraph.git
> >> cd giraph
> >> git revert ac93c3b6c5bce5f22b293b29df91663ca7d7ce63
> >> mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -DskipTests compile
> >>
> >> With that I got valid jars which allowed me to run the
> >> SimpleShortestPaths
> >>
> >> example in my single-node setup Hadoop 2.2.0 + ZooKeeper 3.4.5:
> >> mkdir test_example
> >> ln -s
> >> $GIRAPH_PREFIX/giraph-examples/target/giraph-examples-1.1.0-
> >>
> >> SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> >>
> >> $HADOOP_PREFIX/bin/hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-
> >>
> >> hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner
> >> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> >> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> >> -vip
> >> giraph/input/tiny_graph.txt -vof
> >> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> >> giraph/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca
> >> giraph.zkList=localhost:2181 -yj giraph-examples-1.1.0-SNAPSHOT-for-
> >> hadoop-2.2.0-jar-with-dependencies.jar
> >>
> >> ###########
> >> Notes:
> >> * Wasn't able to do mvn package as it complained about not having the
> >> dependencies giraph-hive, giraph-hbase, etc... I suspect this is just a
> >> matter
> >> of changing the main pom.xml to include those modules with the hadoop
> >> 2.2.0
> >> target or remove those dependencies from the giraph-dist if they no
> >> longer
> >> apply.
> >> * You may get some memory limit exceeded problems for the
> >> ApplicationMaster.
> >> To fix, either increase the yarn.nodemanager.vmem-pmem-ratio parameter in
> >> yarn-
> >> site.xml or apply my patch to prevent allocation of heap size equal to
> >> total
> >> memory size, described at:
> >> https://issues.apache.org/jira/browse/GIRAPH-812
> >> * In my case, the application didn't exit even after outputting the
> >> result and
> >> having recognized that all nodes voted to terminate. I've opened an issue
> >> in JIRA regarding this so if you suffer from the same, you can apply that
> >> patch
> >> too: https://issues.apache.org/jira/browse/GIRAPH-811
> >>
> >> ###########
> >> Relevant JIRA issues:
> >> * https://issues.apache.org/jira/browse/GIRAPH-798
> >> * https://issues.apache.org/jira/browse/GIRAPH-804
> >> * https://issues.apache.org/jira/browse/GIRAPH-811
> >> * https://issues.apache.org/jira/browse/GIRAPH-812
> >>
> >> Best regards,
> >> Alexandre
> >>
> >> On Friday 06 December 2013 00:32:32 Yong Guo wrote:
> >> > Hi,
> >> >
> >> > I also got this error when tried to build Giraph on top of hadoop-2.2.0
> >> > using similar commands. Can anyone help?
> >> >
> >> > Thanks,
> >> > Yong
> >> >
> >> >
> >> > 2013/12/6 Rohit Bhat <ro...@gmail.com>
> >> >
> >> > > I already have yarn (hadoop_2.2.0) on my machine. I tried the
> >>
> >> following 2
> >>
> >> > > commands and got similar errors in both the cases:
> >> > >
> >> > > 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package
> >>
> >> -DskipTests"
> >>
> >> > > 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install
> >> > > package"
> >> > >
> >> > >
> >> > > [INFO] Checking dependency versions
> >> > >
> >> > > [ERROR] Found a problem with the direct dependency
> >> > > commons-httpclient:commons-httpclient of the current project
> >> > >
> >> > > Expected version is 3.0.1
> >> > >
> >> > > Resolved version is 3.0.1
> >>
> >> > > Version 3.1 was expected by artifact:
> >> org.apache.hadoop:hadoop-common
> >>
> >> > > [ERROR] Found a problem with the dependency commons-lang:commons-lang
> >> > >
> >> > > Resolved version is 2.4
> >> > >
> >> > > Version 2.4 was expected by artifact:
> >> > > commons-configuration:commons-configuration
> >> > >
> >> > > Version 2.5 was expected by artifact:
> >> org.apache.hadoop:hadoop-common
> >>
> >> > > [ERROR] Found a problem with the dependency
> >> > > commons-logging:commons-logging
> >> > >
> >> > > Resolved version is 1.0.4
> >>
> >> > > Version 1.1.1 was expected by artifact:
> >> org.apache.hadoop:hadoop-common
> >>
> >> > > [ERROR] Found a problem with the dependency io.netty:netty
> >> > >
> >> > > Resolved version is 3.5.3.Final
> >> > >
> >> > > Version 3.6.2.Final was expected by artifacts: org.apache.
> >> > >
> >> > > hadoop:hadoop-mapreduce-client-common, org.apache.
> >> > > hadoop:hadoop-mapreduce-client-core
> >> > >
> >> > > [ERROR] Found a problem with the dependency
> >>
> >> org.apache.zookeeper:zookeeper
> >>
> >> > > Resolved version is 3.3.3
> >> > >
> >> > > Version 3.4.5 was expected by artifacts:
> >> > > org.apache.hadoop:hadoop-common,
> >> > >
> >> > > org.apache.hadoop:hadoop-mapreduce-client-common
> >> > >
> >> > >
> >> > > Am i missing something?
> >> > >
> >> > > On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar
> >>
> >> <am...@gmail.com>wrote:
> >> > >> Yes. Use this:
> >> > >>
> >> > >> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package
> >> > >>
> >> > >> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang
> >>
> >> <ma...@gmail.com>wrote:
> >> > >>> I get the same error when I compile Giraph against the default
> >>
> >> hadoop
> >>
> >> > >>> version (0.20.203.0), but my project that uses Giraph uses a more
> >> > >>> recent hadoop version. Did you set the hadoop version via a maven
> >> > >>> profile
> >> > >>> when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop
> >>
> >> 1.0)?
> >>
> >> > >>> Presumably, there's a way to compile Giraph for Yarn. There's even
> >> > >>> a
> >> > >>> Yarn profile in the pom.xml. I personally haven't had success
> >>
> >> building
> >>
> >> > >>> it
> >> > >>> though.
> >> > >>>
> >> > >>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat
> >>
> >> <ro...@gmail.com>wrote:
> >> > >>>> Hi,
> >> > >>>>
> >> > >>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have
> >> > >>>> installed Giraph on top of it. I have been trying to
> >> > >>>> execute SimpleShortestPathsComputation example but i am getting
> >> > >>>> the
> >> > >>>> error:
> >>
> >> > >>>> "java.lang.IllegalArgumentException:
> >> checkLocalJobRunnerConfiguration:
> >> > >>>> When using LocalJobRunner, you cannot run in split master / worker
> >>
> >> mode
> >>
> >> > >>>> since there is only 1 task at a time!"
> >> > >>>>
> >> > >>>> I fixed this by adding an entry in mapred-site.xml for
> >> > >>>> mapreduce.jobtracker.address but now i am getting a different
> >>
> >> error:
> >> > >>>> *Exception in thread "main"
> >> > >>>> java.lang.IncompatibleClassChangeError:
> >> > >>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class
> >>
> >> was
> >>
> >> > >>>> expected*
> >> > >>>>
> >> > >>>> at
> >>
> >> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.
> >>
> >> > >>>> java:43) at
> >>
> >> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:4
> >>
> >> > >>>> 56)
> >> > >>>>
> >> > >>>> at
> >>
> >> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter
> >>
> >> > >>>> .java:342) at
> >> > >>>> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
> >> > >>>>
> >> > >>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
> >> > >>>>
> >> > >>>> at java.security.AccessController.doPrivileged(Native Method)
> >> > >>>>
> >> > >>>> at javax.security.auth.Subject.doAs(Subject.java:415)
> >> > >>>>
> >> > >>>> at
> >>
> >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
> >>
> >> > >>>> on.java:1491)>>>>
> >> > >>>>
> >> > >>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
> >> > >>>>
> >> > >>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
> >> > >>>>
> >> > >>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
> >> > >>>>
> >> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >> > >>>>
> >> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> >> > >>>>
> >> > >>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
> >> > >>>>
> >> > >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> > >>>>
> >> > >>>> at
> >>
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
> >>
> >> > >>>> va:57)>>>>
> >> > >>>>
> >> > >>>> at
> >>
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> >>
> >> > >>>> rImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606)
> >> > >>>>
> >> > >>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> >> > >>>>
> >> > >>>> *Is Giraph compatible with yarn? If yes, am i missing anything?*
> >> > >>>>
> >> > >>>> Thanks,
> >> > >>>> Rohit
Re: Issue while executing SimpleShortestPathsComputation
Posted by Rohit Bhat <ro...@gmail.com>.
Hi Eli/Alexandre,
I have tried with Giraph-1.1.0-SNAPSHOT and hadoop-2.2.0 as well using the
steps mentioned by Alexandre, but again i am getting the same error while
running the example:
Command:
$HADOOP_HOME/bin/hadoop jar
giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /giraphInput/tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca
giraph.zkList=localhost:2181 -yj
giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
ERROR yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
state, diagnostics show: Application application_1386601013698_0005 failed
2 times due to AM Container for appattempt_1386601013698_0005_000002 exited
with exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
.Failing this attempt.. Failing the application.
I checked the Hadoop logs as well but couldn't find anything meaningful
there too.
Thanks,
Rohit
On Sun, Dec 8, 2013 at 12:02 PM, Eli Reisman <ap...@gmail.com>wrote:
> Hey all,
>
> So. I think from more recent emails Rohit got this figured out, but, a
> couple points just so they out there again:
>
> Giraph 1.0.x can be built on a "pure YARN" profile with Hadoop
> 2.0.03-alpha or (with a patch thats still floating around on GIRAPH-687
> ticket) against Hadoop 2.0.4 and 2.0.5 alpha.
>
> Much better way to go is to build against Hadoop 2.2.x line using Giraph
> 1.1.x's YARN profile. This features a stable API, some basic support for
> security, and a concurrency fix that could bite you when launching many
> workers.
>
> Finally, you _can_ run Giraph on MRv2 on YARN in which case you need to
> build against a Hadoop 2.x Giraph Maven profile but _not_ the YARN profile.
> This runs on top of MapReduce just as Giraph on Hadoop 1.x or 0.20.x do,
> but riding on Mapper task instead of running the workers in YARN containers.
>
>
>
> On Sat, Dec 7, 2013 at 8:53 PM, Alexandre Fonseca <
> alexandrejorgefonseca@gmail.com> wrote:
>
>> Hello Yong,
>>
>> I was able to compile and run Giraph 1.1.0-SNAPSHOT on top of
>> hadoop-2.2.0.
>> Here's what I did:
>>
>> git clone https://git-wip-us.apache.org/repos/asf/giraph.git
>> cd giraph
>> git revert ac93c3b6c5bce5f22b293b29df91663ca7d7ce63
>> mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -DskipTests compile
>>
>> With that I got valid jars which allowed me to run the SimpleShortestPaths
>> example in my single-node setup Hadoop 2.2.0 + ZooKeeper 3.4.5:
>>
>> mkdir test_example
>> ln -s $GIRAPH_PREFIX/giraph-examples/target/giraph-examples-1.1.0-
>> SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>> $HADOOP_PREFIX/bin/hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-
>> hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner
>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> -vip
>> giraph/input/tiny_graph.txt -vof
>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> giraph/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca
>> giraph.zkList=localhost:2181 -yj giraph-examples-1.1.0-SNAPSHOT-for-
>> hadoop-2.2.0-jar-with-dependencies.jar
>>
>> ###########
>> Notes:
>> * Wasn't able to do mvn package as it complained about not having the
>> dependencies giraph-hive, giraph-hbase, etc... I suspect this is just a
>> matter
>> of changing the main pom.xml to include those modules with the hadoop
>> 2.2.0
>> target or remove those dependencies from the giraph-dist if they no longer
>> apply.
>> * You may get some memory limit exceeded problems for the
>> ApplicationMaster.
>> To fix, either increase the yarn.nodemanager.vmem-pmem-ratio parameter in
>> yarn-
>> site.xml or apply my patch to prevent allocation of heap size equal to
>> total
>> memory size, described at:
>> https://issues.apache.org/jira/browse/GIRAPH-812
>> * In my case, the application didn't exit even after outputting the
>> result and
>> having recognized that all nodes voted to terminate. I've opened an issue
>> in JIRA regarding this so if you suffer from the same, you can apply that
>> patch
>> too: https://issues.apache.org/jira/browse/GIRAPH-811
>>
>> ###########
>> Relevant JIRA issues:
>> * https://issues.apache.org/jira/browse/GIRAPH-798
>> * https://issues.apache.org/jira/browse/GIRAPH-804
>> * https://issues.apache.org/jira/browse/GIRAPH-811
>> * https://issues.apache.org/jira/browse/GIRAPH-812
>>
>> Best regards,
>> Alexandre
>>
>> On Friday 06 December 2013 00:32:32 Yong Guo wrote:
>> > Hi,
>> >
>> > I also got this error when tried to build Giraph on top of hadoop-2.2.0
>> > using similar commands. Can anyone help?
>> >
>> > Thanks,
>> > Yong
>> >
>> >
>> > 2013/12/6 Rohit Bhat <ro...@gmail.com>
>> >
>> > > I already have yarn (hadoop_2.2.0) on my machine. I tried the
>> following 2
>> > > commands and got similar errors in both the cases:
>> > >
>> > > 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package
>> -DskipTests"
>> > >
>> > >
>> > > 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install
>> > > package"
>> > >
>> > >
>> > > [INFO] Checking dependency versions
>> > >
>> > > [ERROR] Found a problem with the direct dependency
>> > > commons-httpclient:commons-httpclient of the current project
>> > >
>> > > Expected version is 3.0.1
>> > >
>> > > Resolved version is 3.0.1
>> > >
>> > > Version 3.1 was expected by artifact:
>> org.apache.hadoop:hadoop-common
>> > >
>> > > [ERROR] Found a problem with the dependency commons-lang:commons-lang
>> > >
>> > > Resolved version is 2.4
>> > >
>> > > Version 2.4 was expected by artifact:
>> > > commons-configuration:commons-configuration
>> > >
>> > > Version 2.5 was expected by artifact:
>> org.apache.hadoop:hadoop-common
>> > >
>> > > [ERROR] Found a problem with the dependency
>> > > commons-logging:commons-logging
>> > >
>> > > Resolved version is 1.0.4
>> > >
>> > > Version 1.1.1 was expected by artifact:
>> org.apache.hadoop:hadoop-common
>> > >
>> > > [ERROR] Found a problem with the dependency io.netty:netty
>> > >
>> > > Resolved version is 3.5.3.Final
>> > >
>> > > Version 3.6.2.Final was expected by artifacts: org.apache.
>> > >
>> > > hadoop:hadoop-mapreduce-client-common, org.apache.
>> > > hadoop:hadoop-mapreduce-client-core
>> > >
>> > > [ERROR] Found a problem with the dependency
>> org.apache.zookeeper:zookeeper
>> > >
>> > > Resolved version is 3.3.3
>> > >
>> > > Version 3.4.5 was expected by artifacts:
>> > > org.apache.hadoop:hadoop-common,
>> > >
>> > > org.apache.hadoop:hadoop-mapreduce-client-common
>> > >
>> > >
>> > > Am i missing something?
>> > >
>> > > On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar
>> <am...@gmail.com>wrote:
>> > >> Yes. Use this:
>> > >>
>> > >> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package
>> > >>
>> > >> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang
>> <ma...@gmail.com>wrote:
>> > >>> I get the same error when I compile Giraph against the default
>> hadoop
>> > >>> version (0.20.203.0), but my project that uses Giraph uses a more
>> > >>> recent hadoop version. Did you set the hadoop version via a maven
>> > >>> profile
>> > >>> when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop
>> 1.0)?
>> > >>>
>> > >>> Presumably, there's a way to compile Giraph for Yarn. There's even a
>> > >>> Yarn profile in the pom.xml. I personally haven't had success
>> building
>> > >>> it
>> > >>> though.
>> > >>>
>> > >>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat
>> <ro...@gmail.com>wrote:
>> > >>>> Hi,
>> > >>>>
>> > >>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have
>> > >>>> installed Giraph on top of it. I have been trying to
>> > >>>> execute SimpleShortestPathsComputation example but i am getting the
>> > >>>> error:
>> > >>>>
>> > >>>> "java.lang.IllegalArgumentException:
>> checkLocalJobRunnerConfiguration:
>> > >>>> When using LocalJobRunner, you cannot run in split master / worker
>> mode
>> > >>>> since there is only 1 task at a time!"
>> > >>>>
>> > >>>> I fixed this by adding an entry in mapred-site.xml for
>> > >>>> mapreduce.jobtracker.address but now i am getting a different
>> error:
>> > >>>>
>> > >>>> *Exception in thread "main" java.lang.IncompatibleClassChangeError:
>> > >>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class
>> was
>> > >>>> expected*
>> > >>>>
>> > >>>> at
>> > >>>>
>> > >>>>
>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.
>> > >>>> java:43) at
>> > >>>>
>> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:4
>> > >>>> 56)
>> > >>>>
>> > >>>> at
>> > >>>>
>> > >>>>
>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter
>> > >>>> .java:342) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>> > >>>>
>> > >>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>> > >>>>
>> > >>>> at java.security.AccessController.doPrivileged(Native Method)
>> > >>>>
>> > >>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> > >>>>
>> > >>>> at
>> > >>>>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
>> > >>>> on.java:1491)>>>>
>> > >>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>> > >>>>
>> > >>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
>> > >>>>
>> > >>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>> > >>>>
>> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> > >>>>
>> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> > >>>>
>> > >>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>> > >>>>
>> > >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > >>>>
>> > >>>> at
>> > >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
>> > >>>> va:57)>>>>
>> > >>>> at
>> > >>>>
>> > >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
>> > >>>> rImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606)
>> > >>>>
>> > >>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>> > >>>>
>> > >>>> *Is Giraph compatible with yarn? If yes, am i missing anything?*
>> > >>>>
>> > >>>> Thanks,
>> > >>>> Rohit
>>
>
>
Re: Issue while executing SimpleShortestPathsComputation
Posted by Eli Reisman <ap...@gmail.com>.
Hey all,
So. I think from more recent emails Rohit got this figured out, but, a
couple points just so they out there again:
Giraph 1.0.x can be built on a "pure YARN" profile with Hadoop 2.0.03-alpha
or (with a patch thats still floating around on GIRAPH-687 ticket) against
Hadoop 2.0.4 and 2.0.5 alpha.
Much better way to go is to build against Hadoop 2.2.x line using Giraph
1.1.x's YARN profile. This features a stable API, some basic support for
security, and a concurrency fix that could bite you when launching many
workers.
Finally, you _can_ run Giraph on MRv2 on YARN in which case you need to
build against a Hadoop 2.x Giraph Maven profile but _not_ the YARN profile.
This runs on top of MapReduce just as Giraph on Hadoop 1.x or 0.20.x do,
but riding on Mapper task instead of running the workers in YARN containers.
On Sat, Dec 7, 2013 at 8:53 PM, Alexandre Fonseca <
alexandrejorgefonseca@gmail.com> wrote:
> Hello Yong,
>
> I was able to compile and run Giraph 1.1.0-SNAPSHOT on top of hadoop-2.2.0.
> Here's what I did:
>
> git clone https://git-wip-us.apache.org/repos/asf/giraph.git
> cd giraph
> git revert ac93c3b6c5bce5f22b293b29df91663ca7d7ce63
> mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -DskipTests compile
>
> With that I got valid jars which allowed me to run the SimpleShortestPaths
> example in my single-node setup Hadoop 2.2.0 + ZooKeeper 3.4.5:
>
> mkdir test_example
> ln -s $GIRAPH_PREFIX/giraph-examples/target/giraph-examples-1.1.0-
> SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> $HADOOP_PREFIX/bin/hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-
> hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip
> giraph/input/tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> giraph/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca
> giraph.zkList=localhost:2181 -yj giraph-examples-1.1.0-SNAPSHOT-for-
> hadoop-2.2.0-jar-with-dependencies.jar
>
> ###########
> Notes:
> * Wasn't able to do mvn package as it complained about not having the
> dependencies giraph-hive, giraph-hbase, etc... I suspect this is just a
> matter
> of changing the main pom.xml to include those modules with the hadoop 2.2.0
> target or remove those dependencies from the giraph-dist if they no longer
> apply.
> * You may get some memory limit exceeded problems for the
> ApplicationMaster.
> To fix, either increase the yarn.nodemanager.vmem-pmem-ratio parameter in
> yarn-
> site.xml or apply my patch to prevent allocation of heap size equal to
> total
> memory size, described at:
> https://issues.apache.org/jira/browse/GIRAPH-812
> * In my case, the application didn't exit even after outputting the result
> and
> having recognized that all nodes voted to terminate. I've opened an issue
> in JIRA regarding this so if you suffer from the same, you can apply that
> patch
> too: https://issues.apache.org/jira/browse/GIRAPH-811
>
> ###########
> Relevant JIRA issues:
> * https://issues.apache.org/jira/browse/GIRAPH-798
> * https://issues.apache.org/jira/browse/GIRAPH-804
> * https://issues.apache.org/jira/browse/GIRAPH-811
> * https://issues.apache.org/jira/browse/GIRAPH-812
>
> Best regards,
> Alexandre
>
> On Friday 06 December 2013 00:32:32 Yong Guo wrote:
> > Hi,
> >
> > I also got this error when tried to build Giraph on top of hadoop-2.2.0
> > using similar commands. Can anyone help?
> >
> > Thanks,
> > Yong
> >
> >
> > 2013/12/6 Rohit Bhat <ro...@gmail.com>
> >
> > > I already have yarn (hadoop_2.2.0) on my machine. I tried the
> following 2
> > > commands and got similar errors in both the cases:
> > >
> > > 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package
> -DskipTests"
> > >
> > >
> > > 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install
> > > package"
> > >
> > >
> > > [INFO] Checking dependency versions
> > >
> > > [ERROR] Found a problem with the direct dependency
> > > commons-httpclient:commons-httpclient of the current project
> > >
> > > Expected version is 3.0.1
> > >
> > > Resolved version is 3.0.1
> > >
> > > Version 3.1 was expected by artifact: org.apache.hadoop:hadoop-common
> > >
> > > [ERROR] Found a problem with the dependency commons-lang:commons-lang
> > >
> > > Resolved version is 2.4
> > >
> > > Version 2.4 was expected by artifact:
> > > commons-configuration:commons-configuration
> > >
> > > Version 2.5 was expected by artifact: org.apache.hadoop:hadoop-common
> > >
> > > [ERROR] Found a problem with the dependency
> > > commons-logging:commons-logging
> > >
> > > Resolved version is 1.0.4
> > >
> > > Version 1.1.1 was expected by artifact:
> org.apache.hadoop:hadoop-common
> > >
> > > [ERROR] Found a problem with the dependency io.netty:netty
> > >
> > > Resolved version is 3.5.3.Final
> > >
> > > Version 3.6.2.Final was expected by artifacts: org.apache.
> > >
> > > hadoop:hadoop-mapreduce-client-common, org.apache.
> > > hadoop:hadoop-mapreduce-client-core
> > >
> > > [ERROR] Found a problem with the dependency
> org.apache.zookeeper:zookeeper
> > >
> > > Resolved version is 3.3.3
> > >
> > > Version 3.4.5 was expected by artifacts:
> > > org.apache.hadoop:hadoop-common,
> > >
> > > org.apache.hadoop:hadoop-mapreduce-client-common
> > >
> > >
> > > Am i missing something?
> > >
> > > On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar
> <am...@gmail.com>wrote:
> > >> Yes. Use this:
> > >>
> > >> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package
> > >>
> > >> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang
> <ma...@gmail.com>wrote:
> > >>> I get the same error when I compile Giraph against the default hadoop
> > >>> version (0.20.203.0), but my project that uses Giraph uses a more
> > >>> recent hadoop version. Did you set the hadoop version via a maven
> > >>> profile
> > >>> when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop 1.0)?
> > >>>
> > >>> Presumably, there's a way to compile Giraph for Yarn. There's even a
> > >>> Yarn profile in the pom.xml. I personally haven't had success
> building
> > >>> it
> > >>> though.
> > >>>
> > >>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat
> <ro...@gmail.com>wrote:
> > >>>> Hi,
> > >>>>
> > >>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have
> > >>>> installed Giraph on top of it. I have been trying to
> > >>>> execute SimpleShortestPathsComputation example but i am getting the
> > >>>> error:
> > >>>>
> > >>>> "java.lang.IllegalArgumentException:
> checkLocalJobRunnerConfiguration:
> > >>>> When using LocalJobRunner, you cannot run in split master / worker
> mode
> > >>>> since there is only 1 task at a time!"
> > >>>>
> > >>>> I fixed this by adding an entry in mapred-site.xml for
> > >>>> mapreduce.jobtracker.address but now i am getting a different error:
> > >>>>
> > >>>> *Exception in thread "main" java.lang.IncompatibleClassChangeError:
> > >>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class
> was
> > >>>> expected*
> > >>>>
> > >>>> at
> > >>>>
> > >>>>
> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.
> > >>>> java:43) at
> > >>>>
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:4
> > >>>> 56)
> > >>>>
> > >>>> at
> > >>>>
> > >>>>
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter
> > >>>> .java:342) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
> > >>>>
> > >>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
> > >>>>
> > >>>> at java.security.AccessController.doPrivileged(Native Method)
> > >>>>
> > >>>> at javax.security.auth.Subject.doAs(Subject.java:415)
> > >>>>
> > >>>> at
> > >>>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
> > >>>> on.java:1491)>>>>
> > >>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
> > >>>>
> > >>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
> > >>>>
> > >>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
> > >>>>
> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > >>>>
> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> > >>>>
> > >>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
> > >>>>
> > >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >>>>
> > >>>> at
> > >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
> > >>>> va:57)>>>>
> > >>>> at
> > >>>>
> > >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> > >>>> rImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606)
> > >>>>
> > >>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> > >>>>
> > >>>> *Is Giraph compatible with yarn? If yes, am i missing anything?*
> > >>>>
> > >>>> Thanks,
> > >>>> Rohit
>
Re: Issue while executing SimpleShortestPathsComputation
Posted by Alexandre Fonseca <al...@gmail.com>.
Hello Yong,
I was able to compile and run Giraph 1.1.0-SNAPSHOT on top of hadoop-2.2.0.
Here's what I did:
git clone https://git-wip-us.apache.org/repos/asf/giraph.git
cd giraph
git revert ac93c3b6c5bce5f22b293b29df91663ca7d7ce63
mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -DskipTests compile
With that I got valid jars which allowed me to run the SimpleShortestPaths
example in my single-node setup Hadoop 2.2.0 + ZooKeeper 3.4.5:
mkdir test_example
ln -s $GIRAPH_PREFIX/giraph-examples/target/giraph-examples-1.1.0-
SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
$HADOOP_PREFIX/bin/hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-
hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip
giraph/input/tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
giraph/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca
giraph.zkList=localhost:2181 -yj giraph-examples-1.1.0-SNAPSHOT-for-
hadoop-2.2.0-jar-with-dependencies.jar
###########
Notes:
* Wasn't able to do mvn package as it complained about not having the
dependencies giraph-hive, giraph-hbase, etc... I suspect this is just a matter
of changing the main pom.xml to include those modules with the hadoop 2.2.0
target or remove those dependencies from the giraph-dist if they no longer
apply.
* You may get some memory limit exceeded problems for the ApplicationMaster.
To fix, either increase the yarn.nodemanager.vmem-pmem-ratio parameter in yarn-
site.xml or apply my patch to prevent allocation of heap size equal to total
memory size, described at: https://issues.apache.org/jira/browse/GIRAPH-812
* In my case, the application didn't exit even after outputting the result and
having recognized that all nodes voted to terminate. I've opened an issue
in JIRA regarding this so if you suffer from the same, you can apply that patch
too: https://issues.apache.org/jira/browse/GIRAPH-811
###########
Relevant JIRA issues:
* https://issues.apache.org/jira/browse/GIRAPH-798
* https://issues.apache.org/jira/browse/GIRAPH-804
* https://issues.apache.org/jira/browse/GIRAPH-811
* https://issues.apache.org/jira/browse/GIRAPH-812
Best regards,
Alexandre
On Friday 06 December 2013 00:32:32 Yong Guo wrote:
> Hi,
>
> I also got this error when tried to build Giraph on top of hadoop-2.2.0
> using similar commands. Can anyone help?
>
> Thanks,
> Yong
>
>
> 2013/12/6 Rohit Bhat <ro...@gmail.com>
>
> > I already have yarn (hadoop_2.2.0) on my machine. I tried the following 2
> > commands and got similar errors in both the cases:
> >
> > 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package -DskipTests"
> >
> >
> > 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install
> > package"
> >
> >
> > [INFO] Checking dependency versions
> >
> > [ERROR] Found a problem with the direct dependency
> > commons-httpclient:commons-httpclient of the current project
> >
> > Expected version is 3.0.1
> >
> > Resolved version is 3.0.1
> >
> > Version 3.1 was expected by artifact: org.apache.hadoop:hadoop-common
> >
> > [ERROR] Found a problem with the dependency commons-lang:commons-lang
> >
> > Resolved version is 2.4
> >
> > Version 2.4 was expected by artifact:
> > commons-configuration:commons-configuration
> >
> > Version 2.5 was expected by artifact: org.apache.hadoop:hadoop-common
> >
> > [ERROR] Found a problem with the dependency
> > commons-logging:commons-logging
> >
> > Resolved version is 1.0.4
> >
> > Version 1.1.1 was expected by artifact: org.apache.hadoop:hadoop-common
> >
> > [ERROR] Found a problem with the dependency io.netty:netty
> >
> > Resolved version is 3.5.3.Final
> >
> > Version 3.6.2.Final was expected by artifacts: org.apache.
> >
> > hadoop:hadoop-mapreduce-client-common, org.apache.
> > hadoop:hadoop-mapreduce-client-core
> >
> > [ERROR] Found a problem with the dependency org.apache.zookeeper:zookeeper
> >
> > Resolved version is 3.3.3
> >
> > Version 3.4.5 was expected by artifacts:
> > org.apache.hadoop:hadoop-common,
> >
> > org.apache.hadoop:hadoop-mapreduce-client-common
> >
> >
> > Am i missing something?
> >
> > On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar
<am...@gmail.com>wrote:
> >> Yes. Use this:
> >>
> >> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package
> >>
> >> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang
<ma...@gmail.com>wrote:
> >>> I get the same error when I compile Giraph against the default hadoop
> >>> version (0.20.203.0), but my project that uses Giraph uses a more
> >>> recent hadoop version. Did you set the hadoop version via a maven
> >>> profile
> >>> when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop 1.0)?
> >>>
> >>> Presumably, there's a way to compile Giraph for Yarn. There's even a
> >>> Yarn profile in the pom.xml. I personally haven't had success building
> >>> it
> >>> though.
> >>>
> >>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat
<ro...@gmail.com>wrote:
> >>>> Hi,
> >>>>
> >>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have
> >>>> installed Giraph on top of it. I have been trying to
> >>>> execute SimpleShortestPathsComputation example but i am getting the
> >>>> error:
> >>>>
> >>>> "java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration:
> >>>> When using LocalJobRunner, you cannot run in split master / worker mode
> >>>> since there is only 1 task at a time!"
> >>>>
> >>>> I fixed this by adding an entry in mapred-site.xml for
> >>>> mapreduce.jobtracker.address but now i am getting a different error:
> >>>>
> >>>> *Exception in thread "main" java.lang.IncompatibleClassChangeError:
> >>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
> >>>> expected*
> >>>>
> >>>> at
> >>>>
> >>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.
> >>>> java:43) at
> >>>> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:4
> >>>> 56)
> >>>>
> >>>> at
> >>>>
> >>>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter
> >>>> .java:342) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
> >>>>
> >>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
> >>>>
> >>>> at java.security.AccessController.doPrivileged(Native Method)
> >>>>
> >>>> at javax.security.auth.Subject.doAs(Subject.java:415)
> >>>>
> >>>> at
> >>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
> >>>> on.java:1491)>>>>
> >>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
> >>>>
> >>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
> >>>>
> >>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
> >>>>
> >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >>>>
> >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> >>>>
> >>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
> >>>>
> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>>
> >>>> at
> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
> >>>> va:57)>>>>
> >>>> at
> >>>>
> >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> >>>> rImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606)
> >>>>
> >>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> >>>>
> >>>> *Is Giraph compatible with yarn? If yes, am i missing anything?*
> >>>>
> >>>> Thanks,
> >>>> Rohit
Re: Issue while executing SimpleShortestPathsComputation
Posted by Rohit Bhat <ro...@gmail.com>.
I am using Giraph 1.0.0 and hadoop-2.0.3-alpha now. Compilation works but
when i run the SimpleShortestPathsVertex example, I am getting the error:
ERROR yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsVertex reports FAILED state,
diagnostics show: Application application_1386306822242_0007 failed 1 times
due to AM Container for appattempt_1386306822242_0007_000001 exited with
exitCode: 1 due to:
.Failing this attempt.. Failing the application.
Any idea what is causing this?
On Thu, Dec 5, 2013 at 5:58 PM, Rohit Bhat <ro...@gmail.com> wrote:
> Can someone please confirm if giraph 1.0.0 works with hadoop-2.2.0?
>
>
> On Thu, Dec 5, 2013 at 5:58 PM, Rohit Bhat <ro...@gmail.com>wrote:
>
>> I am using giraph 1.0.0 but with yarn (hadoop 2.2.0)
>>
>>
>> On Thu, Dec 5, 2013 at 5:49 PM, Yong Guo <gu...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I think it might be the Giraph version problem. When I tried to build
>>> giraph*1.0.0* on hadoop2.0.3-alpha, I can compile successfully by "*mvn
>>> -Phadoop_yarn -Dhadoop.version=2.0.3-alpha clean package -DskipTests*".
>>>
>>> The command failed when I used giraph*1.1.0*.
>>>
>>> Thanks,
>>> Yong
>>>
>>>
>>>
>>> 2013/12/6 Yong Guo <gu...@gmail.com>
>>>
>>>> Hi,
>>>>
>>>> I also got this error when tried to build Giraph on top of hadoop-2.2.0
>>>> using similar commands. Can anyone help?
>>>>
>>>> Thanks,
>>>> Yong
>>>>
>>>>
>>>> 2013/12/6 Rohit Bhat <ro...@gmail.com>
>>>>
>>>>> I already have yarn (hadoop_2.2.0) on my machine. I tried the
>>>>> following 2 commands and got similar errors in both the cases:
>>>>>
>>>>> 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package -
>>>>> DskipTests"
>>>>>
>>>>> 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install
>>>>> package"
>>>>>
>>>>>
>>>>> [INFO] Checking dependency versions
>>>>>
>>>>> [ERROR] Found a problem with the direct dependency
>>>>> commons-httpclient:commons-httpclient of the current project
>>>>>
>>>>> Expected version is 3.0.1
>>>>>
>>>>> Resolved version is 3.0.1
>>>>>
>>>>> Version 3.1 was expected by artifact: org.apache.
>>>>> hadoop:hadoop-common
>>>>>
>>>>> [ERROR] Found a problem with the dependency commons-lang:commons-lang
>>>>>
>>>>> Resolved version is 2.4
>>>>>
>>>>> Version 2.4 was expected by artifact:
>>>>> commons-configuration:commons-configuration
>>>>>
>>>>> Version 2.5 was expected by artifact: org.apache.
>>>>> hadoop:hadoop-common
>>>>>
>>>>> [ERROR] Found a problem with the dependency
>>>>> commons-logging:commons-logging
>>>>>
>>>>> Resolved version is 1.0.4
>>>>>
>>>>> Version 1.1.1 was expected by artifact: org.apache.
>>>>> hadoop:hadoop-common
>>>>>
>>>>> [ERROR] Found a problem with the dependency io.netty:netty
>>>>>
>>>>> Resolved version is 3.5.3.Final
>>>>>
>>>>> Version 3.6.2.Final was expected by artifacts: org.apache.
>>>>> hadoop:hadoop-mapreduce-client-common, org.apache.
>>>>> hadoop:hadoop-mapreduce-client-core
>>>>>
>>>>> [ERROR] Found a problem with the dependency org.apache.
>>>>> zookeeper:zookeeper
>>>>>
>>>>> Resolved version is 3.3.3
>>>>>
>>>>> Version 3.4.5 was expected by artifacts: org.apache.
>>>>> hadoop:hadoop-common, org.apache.hadoop:hadoop-mapreduce-client-common
>>>>>
>>>>>
>>>>>
>>>>> Am i missing something?
>>>>>
>>>>>
>>>>> On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar <
>>>>> ameya.vilankar@gmail.com> wrote:
>>>>>
>>>>>> Yes. Use this:
>>>>>>
>>>>>> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package
>>>>>>
>>>>>>
>>>>>> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang <manuellagang@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> I get the same error when I compile Giraph against the default
>>>>>>> hadoop version (0.20.203.0), but my project that uses Giraph uses a
>>>>>>> more recent hadoop version. Did you set the hadoop version via a maven
>>>>>>> profile when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for
>>>>>>> hadoop 1.0)?
>>>>>>>
>>>>>>> Presumably, there's a way to compile Giraph for Yarn. There's even a
>>>>>>> Yarn profile in the pom.xml. I personally haven't had success building it
>>>>>>> though.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat <rohitbhat.nitk@gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have
>>>>>>>> installed Giraph on top of it. I have been trying to
>>>>>>>> execute SimpleShortestPathsComputation example but i am getting the error:
>>>>>>>>
>>>>>>>> "java.lang.IllegalArgumentException:
>>>>>>>> checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run
>>>>>>>> in split master / worker mode since there is only 1 task at a time!"
>>>>>>>>
>>>>>>>> I fixed this by adding an entry in mapred-site.xml for
>>>>>>>> mapreduce.jobtracker.address but now i am getting a different error:
>>>>>>>>
>>>>>>>> *Exception in thread "main" java.lang.IncompatibleClassChangeError:
>>>>>>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
>>>>>>>> expected*
>>>>>>>> at
>>>>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>>>>> at
>>>>>>>> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
>>>>>>>> at
>>>>>>>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
>>>>>>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>>>>>>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>>>>>> at
>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>>>>>>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>>>>>>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
>>>>>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>>>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>> at
>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>>> at
>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>>>>>>>
>>>>>>>> *Is Giraph compatible with yarn? If yes, am i missing anything?*
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Rohit
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Issue while executing SimpleShortestPathsComputation
Posted by Rohit Bhat <ro...@gmail.com>.
Can someone please confirm if giraph 1.0.0 works with hadoop-2.2.0?
On Thu, Dec 5, 2013 at 5:58 PM, Rohit Bhat <ro...@gmail.com> wrote:
> I am using giraph 1.0.0 but with yarn (hadoop 2.2.0)
>
>
> On Thu, Dec 5, 2013 at 5:49 PM, Yong Guo <gu...@gmail.com> wrote:
>
>> Hi,
>>
>> I think it might be the Giraph version problem. When I tried to build
>> giraph*1.0.0* on hadoop2.0.3-alpha, I can compile successfully by "*mvn
>> -Phadoop_yarn -Dhadoop.version=2.0.3-alpha clean package -DskipTests*".
>>
>> The command failed when I used giraph*1.1.0*.
>>
>> Thanks,
>> Yong
>>
>>
>>
>> 2013/12/6 Yong Guo <gu...@gmail.com>
>>
>>> Hi,
>>>
>>> I also got this error when tried to build Giraph on top of hadoop-2.2.0
>>> using similar commands. Can anyone help?
>>>
>>> Thanks,
>>> Yong
>>>
>>>
>>> 2013/12/6 Rohit Bhat <ro...@gmail.com>
>>>
>>>> I already have yarn (hadoop_2.2.0) on my machine. I tried the following
>>>> 2 commands and got similar errors in both the cases:
>>>>
>>>> 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package -
>>>> DskipTests"
>>>>
>>>> 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install
>>>> package"
>>>>
>>>>
>>>> [INFO] Checking dependency versions
>>>>
>>>> [ERROR] Found a problem with the direct dependency
>>>> commons-httpclient:commons-httpclient of the current project
>>>>
>>>> Expected version is 3.0.1
>>>>
>>>> Resolved version is 3.0.1
>>>>
>>>> Version 3.1 was expected by artifact: org.apache.hadoop:hadoop-common
>>>>
>>>>
>>>> [ERROR] Found a problem with the dependency commons-lang:commons-lang
>>>>
>>>> Resolved version is 2.4
>>>>
>>>> Version 2.4 was expected by artifact:
>>>> commons-configuration:commons-configuration
>>>>
>>>> Version 2.5 was expected by artifact: org.apache.hadoop:hadoop-common
>>>>
>>>>
>>>> [ERROR] Found a problem with the dependency
>>>> commons-logging:commons-logging
>>>>
>>>> Resolved version is 1.0.4
>>>>
>>>> Version 1.1.1 was expected by artifact: org.apache.
>>>> hadoop:hadoop-common
>>>>
>>>> [ERROR] Found a problem with the dependency io.netty:netty
>>>>
>>>> Resolved version is 3.5.3.Final
>>>>
>>>> Version 3.6.2.Final was expected by artifacts: org.apache.
>>>> hadoop:hadoop-mapreduce-client-common, org.apache.
>>>> hadoop:hadoop-mapreduce-client-core
>>>>
>>>> [ERROR] Found a problem with the dependency org.apache.
>>>> zookeeper:zookeeper
>>>>
>>>> Resolved version is 3.3.3
>>>>
>>>> Version 3.4.5 was expected by artifacts: org.apache.
>>>> hadoop:hadoop-common, org.apache.hadoop:hadoop-mapreduce-client-common
>>>>
>>>>
>>>> Am i missing something?
>>>>
>>>>
>>>> On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar <
>>>> ameya.vilankar@gmail.com> wrote:
>>>>
>>>>> Yes. Use this:
>>>>>
>>>>> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package
>>>>>
>>>>>
>>>>> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang <ma...@gmail.com>wrote:
>>>>>
>>>>>> I get the same error when I compile Giraph against the default hadoop
>>>>>> version (0.20.203.0), but my project that uses Giraph uses a more
>>>>>> recent hadoop version. Did you set the hadoop version via a maven profile
>>>>>> when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop 1.0)?
>>>>>>
>>>>>>
>>>>>> Presumably, there's a way to compile Giraph for Yarn. There's even a
>>>>>> Yarn profile in the pom.xml. I personally haven't had success building it
>>>>>> though.
>>>>>>
>>>>>>
>>>>>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat <ro...@gmail.com>wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have
>>>>>>> installed Giraph on top of it. I have been trying to
>>>>>>> execute SimpleShortestPathsComputation example but i am getting the error:
>>>>>>>
>>>>>>> "java.lang.IllegalArgumentException:
>>>>>>> checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run
>>>>>>> in split master / worker mode since there is only 1 task at a time!"
>>>>>>>
>>>>>>> I fixed this by adding an entry in mapred-site.xml for
>>>>>>> mapreduce.jobtracker.address but now i am getting a different error:
>>>>>>>
>>>>>>> *Exception in thread "main" java.lang.IncompatibleClassChangeError:
>>>>>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
>>>>>>> expected*
>>>>>>> at
>>>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>>>> at
>>>>>>> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
>>>>>>> at
>>>>>>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
>>>>>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>>>>>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>>>>> at
>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>>>>>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>>>>>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
>>>>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>> at
>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>> at
>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>>>>>>
>>>>>>> *Is Giraph compatible with yarn? If yes, am i missing anything?*
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Rohit
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Issue while executing SimpleShortestPathsComputation
Posted by Rohit Bhat <ro...@gmail.com>.
I am using giraph 1.0.0 but with yarn (hadoop 2.2.0)
On Thu, Dec 5, 2013 at 5:49 PM, Yong Guo <gu...@gmail.com> wrote:
> Hi,
>
> I think it might be the Giraph version problem. When I tried to build
> giraph*1.0.0* on hadoop2.0.3-alpha, I can compile successfully by "*mvn
> -Phadoop_yarn -Dhadoop.version=2.0.3-alpha clean package -DskipTests*".
>
> The command failed when I used giraph*1.1.0*.
>
> Thanks,
> Yong
>
>
>
> 2013/12/6 Yong Guo <gu...@gmail.com>
>
>> Hi,
>>
>> I also got this error when tried to build Giraph on top of hadoop-2.2.0
>> using similar commands. Can anyone help?
>>
>> Thanks,
>> Yong
>>
>>
>> 2013/12/6 Rohit Bhat <ro...@gmail.com>
>>
>>> I already have yarn (hadoop_2.2.0) on my machine. I tried the following
>>> 2 commands and got similar errors in both the cases:
>>>
>>> 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package -
>>> DskipTests"
>>>
>>> 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install
>>> package"
>>>
>>>
>>> [INFO] Checking dependency versions
>>>
>>> [ERROR] Found a problem with the direct dependency
>>> commons-httpclient:commons-httpclient of the current project
>>>
>>> Expected version is 3.0.1
>>>
>>> Resolved version is 3.0.1
>>>
>>> Version 3.1 was expected by artifact: org.apache.hadoop:hadoop-common
>>>
>>> [ERROR] Found a problem with the dependency commons-lang:commons-lang
>>>
>>> Resolved version is 2.4
>>>
>>> Version 2.4 was expected by artifact:
>>> commons-configuration:commons-configuration
>>>
>>> Version 2.5 was expected by artifact: org.apache.hadoop:hadoop-common
>>>
>>> [ERROR] Found a problem with the dependency
>>> commons-logging:commons-logging
>>>
>>> Resolved version is 1.0.4
>>>
>>> Version 1.1.1 was expected by artifact: org.apache.
>>> hadoop:hadoop-common
>>>
>>> [ERROR] Found a problem with the dependency io.netty:netty
>>>
>>> Resolved version is 3.5.3.Final
>>>
>>> Version 3.6.2.Final was expected by artifacts: org.apache.
>>> hadoop:hadoop-mapreduce-client-common, org.apache.
>>> hadoop:hadoop-mapreduce-client-core
>>>
>>> [ERROR] Found a problem with the dependency org.apache.
>>> zookeeper:zookeeper
>>>
>>> Resolved version is 3.3.3
>>>
>>> Version 3.4.5 was expected by artifacts: org.apache.
>>> hadoop:hadoop-common, org.apache.hadoop:hadoop-mapreduce-client-common
>>>
>>>
>>> Am i missing something?
>>>
>>>
>>> On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar <ameya.vilankar@gmail.com
>>> > wrote:
>>>
>>>> Yes. Use this:
>>>>
>>>> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package
>>>>
>>>>
>>>> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang <ma...@gmail.com>wrote:
>>>>
>>>>> I get the same error when I compile Giraph against the default hadoop
>>>>> version (0.20.203.0), but my project that uses Giraph uses a more
>>>>> recent hadoop version. Did you set the hadoop version via a maven profile
>>>>> when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop 1.0)?
>>>>>
>>>>> Presumably, there's a way to compile Giraph for Yarn. There's even a
>>>>> Yarn profile in the pom.xml. I personally haven't had success building it
>>>>> though.
>>>>>
>>>>>
>>>>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat <ro...@gmail.com>wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have
>>>>>> installed Giraph on top of it. I have been trying to
>>>>>> execute SimpleShortestPathsComputation example but i am getting the error:
>>>>>>
>>>>>> "java.lang.IllegalArgumentException:
>>>>>> checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run
>>>>>> in split master / worker mode since there is only 1 task at a time!"
>>>>>>
>>>>>> I fixed this by adding an entry in mapred-site.xml for
>>>>>> mapreduce.jobtracker.address but now i am getting a different error:
>>>>>>
>>>>>> *Exception in thread "main" java.lang.IncompatibleClassChangeError:
>>>>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
>>>>>> expected*
>>>>>> at
>>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>>> at
>>>>>> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
>>>>>> at
>>>>>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
>>>>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>>>>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>>>> at
>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>>>>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>>>>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
>>>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at
>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>> at
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>>>>>
>>>>>> *Is Giraph compatible with yarn? If yes, am i missing anything?*
>>>>>>
>>>>>> Thanks,
>>>>>> Rohit
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Issue while executing SimpleShortestPathsComputation
Posted by Yong Guo <gu...@gmail.com>.
Hi,
I think it might be the Giraph version problem. When I tried to build giraph
*1.0.0* on hadoop2.0.3-alpha, I can compile successfully by "*mvn
-Phadoop_yarn -Dhadoop.version=2.0.3-alpha clean package -DskipTests*".
The command failed when I used giraph*1.1.0*.
Thanks,
Yong
2013/12/6 Yong Guo <gu...@gmail.com>
> Hi,
>
> I also got this error when tried to build Giraph on top of hadoop-2.2.0
> using similar commands. Can anyone help?
>
> Thanks,
> Yong
>
>
> 2013/12/6 Rohit Bhat <ro...@gmail.com>
>
>> I already have yarn (hadoop_2.2.0) on my machine. I tried the following 2
>> commands and got similar errors in both the cases:
>>
>> 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package -DskipTests
>> "
>>
>> 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install
>> package"
>>
>>
>> [INFO] Checking dependency versions
>>
>> [ERROR] Found a problem with the direct dependency
>> commons-httpclient:commons-httpclient of the current project
>>
>> Expected version is 3.0.1
>>
>> Resolved version is 3.0.1
>>
>> Version 3.1 was expected by artifact: org.apache.hadoop:hadoop-common
>>
>> [ERROR] Found a problem with the dependency commons-lang:commons-lang
>>
>> Resolved version is 2.4
>>
>> Version 2.4 was expected by artifact:
>> commons-configuration:commons-configuration
>>
>> Version 2.5 was expected by artifact: org.apache.hadoop:hadoop-common
>>
>> [ERROR] Found a problem with the dependency
>> commons-logging:commons-logging
>>
>> Resolved version is 1.0.4
>>
>> Version 1.1.1 was expected by artifact: org.apache.hadoop:hadoop-common
>>
>>
>> [ERROR] Found a problem with the dependency io.netty:netty
>>
>> Resolved version is 3.5.3.Final
>>
>> Version 3.6.2.Final was expected by artifacts: org.apache.
>> hadoop:hadoop-mapreduce-client-common, org.apache.
>> hadoop:hadoop-mapreduce-client-core
>>
>> [ERROR] Found a problem with the dependency org.apache.
>> zookeeper:zookeeper
>>
>> Resolved version is 3.3.3
>>
>> Version 3.4.5 was expected by artifacts: org.apache.
>> hadoop:hadoop-common, org.apache.hadoop:hadoop-mapreduce-client-common
>>
>>
>> Am i missing something?
>>
>>
>> On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar <am...@gmail.com>wrote:
>>
>>> Yes. Use this:
>>>
>>> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package
>>>
>>>
>>> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang <ma...@gmail.com>wrote:
>>>
>>>> I get the same error when I compile Giraph against the default hadoop
>>>> version (0.20.203.0), but my project that uses Giraph uses a more
>>>> recent hadoop version. Did you set the hadoop version via a maven profile
>>>> when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop 1.0)?
>>>>
>>>> Presumably, there's a way to compile Giraph for Yarn. There's even a
>>>> Yarn profile in the pom.xml. I personally haven't had success building it
>>>> though.
>>>>
>>>>
>>>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat <ro...@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have
>>>>> installed Giraph on top of it. I have been trying to
>>>>> execute SimpleShortestPathsComputation example but i am getting the error:
>>>>>
>>>>> "java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration:
>>>>> When using LocalJobRunner, you cannot run in split master / worker mode
>>>>> since there is only 1 task at a time!"
>>>>>
>>>>> I fixed this by adding an entry in mapred-site.xml for
>>>>> mapreduce.jobtracker.address but now i am getting a different error:
>>>>>
>>>>> *Exception in thread "main" java.lang.IncompatibleClassChangeError:
>>>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
>>>>> expected*
>>>>> at
>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>> at
>>>>> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
>>>>> at
>>>>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
>>>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>>>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>>> at
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>>>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>>>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
>>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>>>>
>>>>> *Is Giraph compatible with yarn? If yes, am i missing anything?*
>>>>>
>>>>> Thanks,
>>>>> Rohit
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Issue while executing SimpleShortestPathsComputation
Posted by Yong Guo <gu...@gmail.com>.
Hi,
I also got this error when tried to build Giraph on top of hadoop-2.2.0
using similar commands. Can anyone help?
Thanks,
Yong
2013/12/6 Rohit Bhat <ro...@gmail.com>
> I already have yarn (hadoop_2.2.0) on my machine. I tried the following 2
> commands and got similar errors in both the cases:
>
> 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package -DskipTests"
>
>
> 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install
> package"
>
>
> [INFO] Checking dependency versions
>
> [ERROR] Found a problem with the direct dependency
> commons-httpclient:commons-httpclient of the current project
>
> Expected version is 3.0.1
>
> Resolved version is 3.0.1
>
> Version 3.1 was expected by artifact: org.apache.hadoop:hadoop-common
>
> [ERROR] Found a problem with the dependency commons-lang:commons-lang
>
> Resolved version is 2.4
>
> Version 2.4 was expected by artifact:
> commons-configuration:commons-configuration
>
> Version 2.5 was expected by artifact: org.apache.hadoop:hadoop-common
>
> [ERROR] Found a problem with the dependency
> commons-logging:commons-logging
>
> Resolved version is 1.0.4
>
> Version 1.1.1 was expected by artifact: org.apache.hadoop:hadoop-common
>
> [ERROR] Found a problem with the dependency io.netty:netty
>
> Resolved version is 3.5.3.Final
>
> Version 3.6.2.Final was expected by artifacts: org.apache.
> hadoop:hadoop-mapreduce-client-common, org.apache.
> hadoop:hadoop-mapreduce-client-core
>
> [ERROR] Found a problem with the dependency org.apache.zookeeper:zookeeper
>
>
> Resolved version is 3.3.3
>
> Version 3.4.5 was expected by artifacts: org.apache.hadoop:hadoop-common,
> org.apache.hadoop:hadoop-mapreduce-client-common
>
>
> Am i missing something?
>
>
> On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar <am...@gmail.com>wrote:
>
>> Yes. Use this:
>>
>> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package
>>
>>
>> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang <ma...@gmail.com>wrote:
>>
>>> I get the same error when I compile Giraph against the default hadoop
>>> version (0.20.203.0), but my project that uses Giraph uses a more
>>> recent hadoop version. Did you set the hadoop version via a maven profile
>>> when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop 1.0)?
>>>
>>> Presumably, there's a way to compile Giraph for Yarn. There's even a
>>> Yarn profile in the pom.xml. I personally haven't had success building it
>>> though.
>>>
>>>
>>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat <ro...@gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have
>>>> installed Giraph on top of it. I have been trying to
>>>> execute SimpleShortestPathsComputation example but i am getting the error:
>>>>
>>>> "java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration:
>>>> When using LocalJobRunner, you cannot run in split master / worker mode
>>>> since there is only 1 task at a time!"
>>>>
>>>> I fixed this by adding an entry in mapred-site.xml for
>>>> mapreduce.jobtracker.address but now i am getting a different error:
>>>>
>>>> *Exception in thread "main" java.lang.IncompatibleClassChangeError:
>>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
>>>> expected*
>>>> at
>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>> at
>>>> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
>>>> at
>>>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
>>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>>>
>>>> *Is Giraph compatible with yarn? If yes, am i missing anything?*
>>>>
>>>> Thanks,
>>>> Rohit
>>>>
>>>
>>>
>>
>
Re: Issue while executing SimpleShortestPathsComputation
Posted by Rohit Bhat <ro...@gmail.com>.
I already have yarn (hadoop_2.2.0) on my machine. I tried the following 2
commands and got similar errors in both the cases:
1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package -DskipTests"
2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install package"
[INFO] Checking dependency versions
[ERROR] Found a problem with the direct dependency
commons-httpclient:commons-httpclient of the current project
Expected version is 3.0.1
Resolved version is 3.0.1
Version 3.1 was expected by artifact: org.apache.hadoop:hadoop-common
[ERROR] Found a problem with the dependency commons-lang:commons-lang
Resolved version is 2.4
Version 2.4 was expected by artifact:
commons-configuration:commons-configuration
Version 2.5 was expected by artifact: org.apache.hadoop:hadoop-common
[ERROR] Found a problem with the dependency commons-logging:commons-logging
Resolved version is 1.0.4
Version 1.1.1 was expected by artifact: org.apache.hadoop:hadoop-common
[ERROR] Found a problem with the dependency io.netty:netty
Resolved version is 3.5.3.Final
Version 3.6.2.Final was expected by artifacts: org.apache.
hadoop:hadoop-mapreduce-client-common, org.apache.
hadoop:hadoop-mapreduce-client-core
[ERROR] Found a problem with the dependency org.apache.zookeeper:zookeeper
Resolved version is 3.3.3
Version 3.4.5 was expected by artifacts: org.apache.hadoop:hadoop-common,
org.apache.hadoop:hadoop-mapreduce-client-common
Am i missing something?
On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar <am...@gmail.com>wrote:
> Yes. Use this:
>
> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package
>
>
> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang <ma...@gmail.com>wrote:
>
>> I get the same error when I compile Giraph against the default hadoop
>> version (0.20.203.0), but my project that uses Giraph uses a more recent
>> hadoop version. Did you set the hadoop version via a maven profile when
>> compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop 1.0)?
>>
>> Presumably, there's a way to compile Giraph for Yarn. There's even a Yarn
>> profile in the pom.xml. I personally haven't had success building it though.
>>
>>
>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat <ro...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have
>>> installed Giraph on top of it. I have been trying to
>>> execute SimpleShortestPathsComputation example but i am getting the error:
>>>
>>> "java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration:
>>> When using LocalJobRunner, you cannot run in split master / worker mode
>>> since there is only 1 task at a time!"
>>>
>>> I fixed this by adding an entry in mapred-site.xml for
>>> mapreduce.jobtracker.address but now i am getting a different error:
>>>
>>> *Exception in thread "main" java.lang.IncompatibleClassChangeError:
>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
>>> expected*
>>> at
>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>> at
>>> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
>>> at
>>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>>
>>> *Is Giraph compatible with yarn? If yes, am i missing anything?*
>>>
>>> Thanks,
>>> Rohit
>>>
>>
>>
>
Re: Issue while executing SimpleShortestPathsComputation
Posted by Ameya Vilankar <am...@gmail.com>.
Yes. Use this:
mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package
On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang <ma...@gmail.com>wrote:
> I get the same error when I compile Giraph against the default hadoop
> version (0.20.203.0), but my project that uses Giraph uses a more recent
> hadoop version. Did you set the hadoop version via a maven profile when
> compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop 1.0)?
>
> Presumably, there's a way to compile Giraph for Yarn. There's even a Yarn
> profile in the pom.xml. I personally haven't had success building it though.
>
>
> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat <ro...@gmail.com>wrote:
>
>> Hi,
>>
>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have
>> installed Giraph on top of it. I have been trying to
>> execute SimpleShortestPathsComputation example but i am getting the error:
>>
>> "java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration:
>> When using LocalJobRunner, you cannot run in split master / worker mode
>> since there is only 1 task at a time!"
>>
>> I fixed this by adding an entry in mapred-site.xml for
>> mapreduce.jobtracker.address but now i am getting a different error:
>>
>> *Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
>> interface org.apache.hadoop.mapreduce.JobContext, but class was expected*
>> at
>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>> at
>> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
>> at
>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>
>> *Is Giraph compatible with yarn? If yes, am i missing anything?*
>>
>> Thanks,
>> Rohit
>>
>
>
Re: Issue while executing SimpleShortestPathsComputation
Posted by Manuel Lagang <ma...@gmail.com>.
I get the same error when I compile Giraph against the default hadoop
version (0.20.203.0), but my project that uses Giraph uses a more recent
hadoop version. Did you set the hadoop version via a maven profile when
compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop 1.0)?
Presumably, there's a way to compile Giraph for Yarn. There's even a Yarn
profile in the pom.xml. I personally haven't had success building it though.
On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat <ro...@gmail.com> wrote:
> Hi,
>
> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have
> installed Giraph on top of it. I have been trying to
> execute SimpleShortestPathsComputation example but i am getting the error:
>
> "java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration:
> When using LocalJobRunner, you cannot run in split master / worker mode
> since there is only 1 task at a time!"
>
> I fixed this by adding an entry in mapred-site.xml for
> mapreduce.jobtracker.address but now i am getting a different error:
>
> *Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
> interface org.apache.hadoop.mapreduce.JobContext, but class was expected*
> at
> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
> *Is Giraph compatible with yarn? If yes, am i missing anything?*
>
> Thanks,
> Rohit
>