You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Roberto Gonzalez <ro...@neclab.eu> on 2015/11/05 14:54:40 UTC
Problems to run
org.apache.giraph.examples.SimpleShortestPathsComputation
Hi all again,
After compiling the version 1.1 I found the following bug:
https://issues.apache.org/jira/browse/GIRAPH-859
I applied the patch and disable the permissions in the HDFS (I would
want not to do that... but I can accept it).
but still executing the example as:
hadoop jar giraph-ex.jar org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
shortestpaths -yj giraph-ex.jar -w 1
The program runs for about 10 minutes (the example graph has 5 nodes)
before failing.
the gam-stderr.log file only contains info about SLF4J, and the
gam-stdout.log finish with:
Container exited with a non-zero exit code 143
2015-11-05 14:25:39,340 INFO [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(605)) - After completion of one conatiner. current status is: completedCount :1 containersToLaunch :2 successfulCount :0 failedCount :1
2015-11-05 14:26:13,414 INFO [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(580)) - Got response from RM for container ask, completedCnt=1
2015-11-05 14:26:13,414 INFO [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(583)) - Got container status for containerID=container_1446634690791_0024_01_000003, state=COMPLETE, exitStatus=2, diagnostics=Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 2
2015-11-05 14:26:13,415 INFO [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(603)) - All container compeleted. done = true
2015-11-05 14:26:13,543 INFO [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:run(195)) - Done true
2015-11-05 14:26:13,543 INFO [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:run(207)) - Forcefully terminating executors with done =:true
2015-11-05 14:26:13,543 INFO [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:finish(221)) - Application completed. Stopping running containers
2015-11-05 14:26:13,578 INFO [main] impl.ContainerManagementProtocolProxy (ContainerManagementProtocolProxy.java:mayBeCloseProxy(145)) - Closing proxy : computer62:59272
2015-11-05 14:26:13,579 INFO [main] impl.ContainerManagementProtocolProxy (ContainerManagementProtocolProxy.java:mayBeCloseProxy(145)) - Closing proxy : computer66:45051
2015-11-05 14:26:13,579 INFO [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:finish(226)) - Application completed. Signalling finish to RM
2015-11-05 14:26:13,586 INFO [main] impl.AMRMClientImpl (AMRMClientImpl.java:unregisterApplicationMaster(321)) - Waiting for application to be successfully unregistered.
2015-11-05 14:26:13,688 INFO [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:main(454)) - Giraph Application Master failed. exiting
2015-11-05 14:26:13,688 INFO [AMRM Callback Handler Thread] impl.AMRMClientAsyncImpl (AMRMClientAsyncImpl.java:run(277)) - Interrupted while waiting for queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
Moreover, even when the exception is 1 minute after the program starts,
it last more than 10 minutes to finish.
Do you have any idea??
Thanks.
--
Dr. Roberto Gonzalez
Research Scientist, Networked Systems and Data Analytics Group
NEC Europe Ltd.
NEC Laboratories Europe
Kurfürsten-Anlage 36
D-69115 Heidelberg
phone +49 6221 4342 256
fax +49 6221 4342 155
e-mail: Roberto.Gonzalez@neclab.eu
NEC Europe Ltd | Registered Office: Athene, Odyssey Business Park, West End Road,
London, HA4 6QE, GB | Registered in England 2832014
Re: Problems to run org.apache.giraph.examples.SimpleShortestPathsComputation
Posted by José Luis Larroque <la...@gmail.com>.
Hi again, Roberto!
I used to build it with this line: mvn package -Phadoop_yarn
-Dhadoop.version=2.4.0 -DskipTests.
I don't remember the exact solution that i took, but you have two options
as far as i know:
- Make that method public, by editing those clases manually
- Remove the STATIC_SASL_SYMBOL munge symbol. To do so, change the
following line 1197 of the pom.xml in the giraph parent directory from
<munge.symbols>PURE_YARN,STATIC_SASL_SYMBOL </munge.symbols>
Hope that some of those solutions are helpfull.
Bye!
Jose
2015-11-09 7:48 GMT-03:00 Roberto Gonzalez <ro...@neclab.eu>:
> The issue is that I use Hadoop 2.4.1, I don't think I can upgrade it
> easily and the compilation with hadoop_2 fails:
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.0:compile
> (default-compile) on project giraph-core: Compilation failure:
> Compilation failure:
> [ERROR]
> /usr/local/giraph-1.1.0/giraph-core/src/main/java/org/apache/giraph/comm/netty/SaslNettyServer.java:[113,32]
> getDefaultProperties() has protected access in
> org.apache.hadoop.security.SaslPropertiesResolver
> [ERROR]
> /usr/local/giraph-1.1.0/giraph-core/src/main/java/org/apache/giraph/comm/netty/SaslNettyClient.java:[93,32]
> getDefaultProperties() has protected access in
> org.apache.hadoop.security.SaslPropertiesResolver
>
>
> So, with mapreduce I cannot compile it and with yarn I cannot use it.
>
> Thanks,
> Roberto
>
>
> On Sat, 2015-11-07 at 15:49 -0800, Roman Shaposhnik wrote:
> > Why do you want to run this as YARN app? Why not use
> > good old MapReduce?
> >
> > Thanks,
> > Roman.
> >
> > On Thu, Nov 5, 2015 at 5:54 AM, Roberto Gonzalez
> > <ro...@neclab.eu> wrote:
> > > Hi all again,
> > >
> > > After compiling the version 1.1 I found the following bug:
> > >
> > > https://issues.apache.org/jira/browse/GIRAPH-859
> > >
> > > I applied the patch and disable the permissions in the HDFS (I would
> > > want not to do that... but I can accept it).
> > >
> > > but still executing the example as:
> > >
> > >
> > > hadoop jar giraph-ex.jar org.apache.giraph.GiraphRunner
> > > org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> > > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> > > -vip tiny_graph.txt -vof
> > > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> > > shortestpaths -yj giraph-ex.jar -w 1
> > >
> > >
> > > The program runs for about 10 minutes (the example graph has 5 nodes)
> > > before failing.
> > >
> > > the gam-stderr.log file only contains info about SLF4J, and the
> > > gam-stdout.log finish with:
> > >
> > > Container exited with a non-zero exit code 143
> > >
> > > 2015-11-05 14:25:39,340 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersCompleted(605)) - After
> completion of one conatiner. current status is: completedCount :1
> containersToLaunch :2 successfulCount :0 failedCount :1
> > > 2015-11-05 14:26:13,414 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersCompleted(580)) - Got response
> from RM for container ask, completedCnt=1
> > > 2015-11-05 14:26:13,414 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersCompleted(583)) - Got container
> status for containerID=container_1446634690791_0024_01_000003,
> state=COMPLETE, exitStatus=2, diagnostics=Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException:
> > > org.apache.hadoop.util.Shell$ExitCodeException:
> > > at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
> > > at org.apache.hadoop.util.Shell.run(Shell.java:418)
> > > at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
> > > at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> > > at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
> > > at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > > at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > >
> > > Container exited with a non-zero exit code 2
> > >
> > > 2015-11-05 14:26:13,415 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersCompleted(603)) - All container
> compeleted. done = true
> > > 2015-11-05 14:26:13,543 INFO [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:run(195)) - Done true
> > > 2015-11-05 14:26:13,543 INFO [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:run(207)) - Forcefully terminating executors
> with done =:true
> > > 2015-11-05 14:26:13,543 INFO [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:finish(221)) - Application completed.
> Stopping running containers
> > > 2015-11-05 14:26:13,578 INFO [main]
> impl.ContainerManagementProtocolProxy
> (ContainerManagementProtocolProxy.java:mayBeCloseProxy(145)) - Closing
> proxy : computer62:59272
> > > 2015-11-05 14:26:13,579 INFO [main]
> impl.ContainerManagementProtocolProxy
> (ContainerManagementProtocolProxy.java:mayBeCloseProxy(145)) - Closing
> proxy : computer66:45051
> > > 2015-11-05 14:26:13,579 INFO [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:finish(226)) - Application completed.
> Signalling finish to RM
> > > 2015-11-05 14:26:13,586 INFO [main] impl.AMRMClientImpl
> (AMRMClientImpl.java:unregisterApplicationMaster(321)) - Waiting for
> application to be successfully unregistered.
> > > 2015-11-05 14:26:13,688 INFO [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:main(454)) - Giraph Application Master
> failed. exiting
> > > 2015-11-05 14:26:13,688 INFO [AMRM Callback Handler Thread]
> impl.AMRMClientAsyncImpl (AMRMClientAsyncImpl.java:run(277)) - Interrupted
> while waiting for queue
> > > java.lang.InterruptedException
> > > at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
> > > at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
> > > at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> > > at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
> > >
> > >
> > > Moreover, even when the exception is 1 minute after the program starts,
> > > it last more than 10 minutes to finish.
> > >
> > > Do you have any idea??
> > >
> > > Thanks.
> > >
> > >
> > >
> > >
> > > --
> > > Dr. Roberto Gonzalez
> > > Research Scientist, Networked Systems and Data Analytics Group
> > > NEC Europe Ltd.
> > > NEC Laboratories Europe
> > > Kurfürsten-Anlage 36
> > >
> > > D-69115 Heidelberg
> > >
> > > phone +49 6221 4342 256
> > > fax +49 6221 4342 155
> > > e-mail: Roberto.Gonzalez@neclab.eu
> > >
> > > NEC Europe Ltd | Registered Office: Athene, Odyssey Business Park,
> West End Road,
> > > London, HA4 6QE, GB | Registered in England 2832014
>
> --
> Dr. Roberto Gonzalez
> Research Scientist, Networked Systems and Data Analytics Group
> NEC Europe Ltd.
> NEC Laboratories Europe
> Kurfürsten-Anlage 36
>
> D-69115 Heidelberg
>
> phone +49 6221 4342 256
> fax +49 6221 4342 155
> e-mail: Roberto.Gonzalez@neclab.eu
>
> NEC Europe Ltd | Registered Office: Athene, Odyssey Business Park, West
> End Road,
> London, HA4 6QE, GB | Registered in England 2832014
>
Re: Problems to run
org.apache.giraph.examples.SimpleShortestPathsComputation
Posted by Roberto Gonzalez <ro...@neclab.eu>.
The issue is that I use Hadoop 2.4.1, I don't think I can upgrade it
easily and the compilation with hadoop_2 fails:
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.0:compile
(default-compile) on project giraph-core: Compilation failure:
Compilation failure:
[ERROR] /usr/local/giraph-1.1.0/giraph-core/src/main/java/org/apache/giraph/comm/netty/SaslNettyServer.java:[113,32] getDefaultProperties() has protected access in org.apache.hadoop.security.SaslPropertiesResolver
[ERROR] /usr/local/giraph-1.1.0/giraph-core/src/main/java/org/apache/giraph/comm/netty/SaslNettyClient.java:[93,32] getDefaultProperties() has protected access in org.apache.hadoop.security.SaslPropertiesResolver
So, with mapreduce I cannot compile it and with yarn I cannot use it.
Thanks,
Roberto
On Sat, 2015-11-07 at 15:49 -0800, Roman Shaposhnik wrote:
> Why do you want to run this as YARN app? Why not use
> good old MapReduce?
>
> Thanks,
> Roman.
>
> On Thu, Nov 5, 2015 at 5:54 AM, Roberto Gonzalez
> <ro...@neclab.eu> wrote:
> > Hi all again,
> >
> > After compiling the version 1.1 I found the following bug:
> >
> > https://issues.apache.org/jira/browse/GIRAPH-859
> >
> > I applied the patch and disable the permissions in the HDFS (I would
> > want not to do that... but I can accept it).
> >
> > but still executing the example as:
> >
> >
> > hadoop jar giraph-ex.jar org.apache.giraph.GiraphRunner
> > org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> > -vip tiny_graph.txt -vof
> > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> > shortestpaths -yj giraph-ex.jar -w 1
> >
> >
> > The program runs for about 10 minutes (the example graph has 5 nodes)
> > before failing.
> >
> > the gam-stderr.log file only contains info about SLF4J, and the
> > gam-stdout.log finish with:
> >
> > Container exited with a non-zero exit code 143
> >
> > 2015-11-05 14:25:39,340 INFO [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(605)) - After completion of one conatiner. current status is: completedCount :1 containersToLaunch :2 successfulCount :0 failedCount :1
> > 2015-11-05 14:26:13,414 INFO [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(580)) - Got response from RM for container ask, completedCnt=1
> > 2015-11-05 14:26:13,414 INFO [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(583)) - Got container status for containerID=container_1446634690791_0024_01_000003, state=COMPLETE, exitStatus=2, diagnostics=Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
> > org.apache.hadoop.util.Shell$ExitCodeException:
> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
> > at org.apache.hadoop.util.Shell.run(Shell.java:418)
> > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
> > at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> > at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
> > at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> >
> > Container exited with a non-zero exit code 2
> >
> > 2015-11-05 14:26:13,415 INFO [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(603)) - All container compeleted. done = true
> > 2015-11-05 14:26:13,543 INFO [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:run(195)) - Done true
> > 2015-11-05 14:26:13,543 INFO [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:run(207)) - Forcefully terminating executors with done =:true
> > 2015-11-05 14:26:13,543 INFO [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:finish(221)) - Application completed. Stopping running containers
> > 2015-11-05 14:26:13,578 INFO [main] impl.ContainerManagementProtocolProxy (ContainerManagementProtocolProxy.java:mayBeCloseProxy(145)) - Closing proxy : computer62:59272
> > 2015-11-05 14:26:13,579 INFO [main] impl.ContainerManagementProtocolProxy (ContainerManagementProtocolProxy.java:mayBeCloseProxy(145)) - Closing proxy : computer66:45051
> > 2015-11-05 14:26:13,579 INFO [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:finish(226)) - Application completed. Signalling finish to RM
> > 2015-11-05 14:26:13,586 INFO [main] impl.AMRMClientImpl (AMRMClientImpl.java:unregisterApplicationMaster(321)) - Waiting for application to be successfully unregistered.
> > 2015-11-05 14:26:13,688 INFO [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:main(454)) - Giraph Application Master failed. exiting
> > 2015-11-05 14:26:13,688 INFO [AMRM Callback Handler Thread] impl.AMRMClientAsyncImpl (AMRMClientAsyncImpl.java:run(277)) - Interrupted while waiting for queue
> > java.lang.InterruptedException
> > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
> > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
> > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> > at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
> >
> >
> > Moreover, even when the exception is 1 minute after the program starts,
> > it last more than 10 minutes to finish.
> >
> > Do you have any idea??
> >
> > Thanks.
> >
> >
> >
> >
> > --
> > Dr. Roberto Gonzalez
> > Research Scientist, Networked Systems and Data Analytics Group
> > NEC Europe Ltd.
> > NEC Laboratories Europe
> > Kurfürsten-Anlage 36
> >
> > D-69115 Heidelberg
> >
> > phone +49 6221 4342 256
> > fax +49 6221 4342 155
> > e-mail: Roberto.Gonzalez@neclab.eu
> >
> > NEC Europe Ltd | Registered Office: Athene, Odyssey Business Park, West End Road,
> > London, HA4 6QE, GB | Registered in England 2832014
--
Dr. Roberto Gonzalez
Research Scientist, Networked Systems and Data Analytics Group
NEC Europe Ltd.
NEC Laboratories Europe
Kurfürsten-Anlage 36
D-69115 Heidelberg
phone +49 6221 4342 256
fax +49 6221 4342 155
e-mail: Roberto.Gonzalez@neclab.eu
NEC Europe Ltd | Registered Office: Athene, Odyssey Business Park, West End Road,
London, HA4 6QE, GB | Registered in England 2832014
Re: Problems to run org.apache.giraph.examples.SimpleShortestPathsComputation
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
Why do you want to run this as YARN app? Why not use
good old MapReduce?
Thanks,
Roman.
On Thu, Nov 5, 2015 at 5:54 AM, Roberto Gonzalez
<ro...@neclab.eu> wrote:
> Hi all again,
>
> After compiling the version 1.1 I found the following bug:
>
> https://issues.apache.org/jira/browse/GIRAPH-859
>
> I applied the patch and disable the permissions in the HDFS (I would
> want not to do that... but I can accept it).
>
> but still executing the example as:
>
>
> hadoop jar giraph-ex.jar org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> shortestpaths -yj giraph-ex.jar -w 1
>
>
> The program runs for about 10 minutes (the example graph has 5 nodes)
> before failing.
>
> the gam-stderr.log file only contains info about SLF4J, and the
> gam-stdout.log finish with:
>
> Container exited with a non-zero exit code 143
>
> 2015-11-05 14:25:39,340 INFO [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(605)) - After completion of one conatiner. current status is: completedCount :1 containersToLaunch :2 successfulCount :0 failedCount :1
> 2015-11-05 14:26:13,414 INFO [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(580)) - Got response from RM for container ask, completedCnt=1
> 2015-11-05 14:26:13,414 INFO [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(583)) - Got container status for containerID=container_1446634690791_0024_01_000003, state=COMPLETE, exitStatus=2, diagnostics=Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
> org.apache.hadoop.util.Shell$ExitCodeException:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Container exited with a non-zero exit code 2
>
> 2015-11-05 14:26:13,415 INFO [AMRM Callback Handler Thread] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:onContainersCompleted(603)) - All container compeleted. done = true
> 2015-11-05 14:26:13,543 INFO [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:run(195)) - Done true
> 2015-11-05 14:26:13,543 INFO [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:run(207)) - Forcefully terminating executors with done =:true
> 2015-11-05 14:26:13,543 INFO [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:finish(221)) - Application completed. Stopping running containers
> 2015-11-05 14:26:13,578 INFO [main] impl.ContainerManagementProtocolProxy (ContainerManagementProtocolProxy.java:mayBeCloseProxy(145)) - Closing proxy : computer62:59272
> 2015-11-05 14:26:13,579 INFO [main] impl.ContainerManagementProtocolProxy (ContainerManagementProtocolProxy.java:mayBeCloseProxy(145)) - Closing proxy : computer66:45051
> 2015-11-05 14:26:13,579 INFO [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:finish(226)) - Application completed. Signalling finish to RM
> 2015-11-05 14:26:13,586 INFO [main] impl.AMRMClientImpl (AMRMClientImpl.java:unregisterApplicationMaster(321)) - Waiting for application to be successfully unregistered.
> 2015-11-05 14:26:13,688 INFO [main] yarn.GiraphApplicationMaster (GiraphApplicationMaster.java:main(454)) - Giraph Application Master failed. exiting
> 2015-11-05 14:26:13,688 INFO [AMRM Callback Handler Thread] impl.AMRMClientAsyncImpl (AMRMClientAsyncImpl.java:run(277)) - Interrupted while waiting for queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
>
>
> Moreover, even when the exception is 1 minute after the program starts,
> it last more than 10 minutes to finish.
>
> Do you have any idea??
>
> Thanks.
>
>
>
>
> --
> Dr. Roberto Gonzalez
> Research Scientist, Networked Systems and Data Analytics Group
> NEC Europe Ltd.
> NEC Laboratories Europe
> Kurfürsten-Anlage 36
>
> D-69115 Heidelberg
>
> phone +49 6221 4342 256
> fax +49 6221 4342 155
> e-mail: Roberto.Gonzalez@neclab.eu
>
> NEC Europe Ltd | Registered Office: Athene, Odyssey Business Park, West End Road,
> London, HA4 6QE, GB | Registered in England 2832014