Cannot run shortest path on Hadoop 2.2

[John Yost <so...@gmail.com>]

Hi Everyone,

The shortest path example fails on my Hadoop 2.2.0 single node cluster, and
I don't see an identifiable root exception.  I am able to execute my
Map/Reduce jobs, including ones that use Accumulo for a source and/or sink,
but cannot get the Giraph example jobs nor my custom Giraph jobs to run.

I followed the build and job launch instructions from the following URL:
http://mail-archives.apache.org/mod_mbox/giraph-user/201312.mbox/%3C1647021.5fbjhLDxPK@chronos7%3E

Here's the Hadoop console output I get when I attempt to run shortest path:

2014-06-16 07:10:09,631 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:main(421)) - Starting GitaphAM
2014-06-16 07:10:10,277 WARN  [main] util.NativeCodeLoader
(NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
2014-06-16 07:10:11,063 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:<init>(168)) - GiraphAM  for ContainerId
container_1402830191668_0017_01_000001 ApplicationAttemptId
appattempt_1402830191668_0017_000001
2014-06-16 07:10:11,130 INFO  [main] client.RMProxy
(RMProxy.java:createRMProxy(56)) - Connecting to ResourceManager at /
0.0.0.0:8030
2014-06-16 07:10:11,136 INFO  [main] impl.NMClientAsyncImpl
(NMClientAsyncImpl.java:serviceInit(107)) - Upper bound of the thread pool
size is 500
2014-06-16 07:10:11,136 INFO  [main] impl.ContainerManagementProtocolProxy
(ContainerManagementProtocolProxy.java:<init>(71)) -
yarn.client.max-nodemanagers-proxies : 500
2014-06-16 07:10:11,299 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested
container ask: Capability[<memory:1024, vCores:0>]Priority[10]
2014-06-16 07:10:11,304 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested
container ask: Capability[<memory:1024, vCores:0>]Priority[10]
2014-06-16 07:10:11,305 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:run(185)) - Wait to finish ..
2014-06-16 07:10:13,331 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersAllocated(605)) - Got response
from RM for container ask, allocatedCnt=1
2014-06-16 07:10:13,331 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersAllocated(608)) - Total allocated
# of container so far : 1 allocated out of 2 required.
2014-06-16 07:10:13,332 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) -
Launching command on a new container.,
containerId=container_1402830191668_0017_01_000002,
containerNode=localhost.localdomain:38256,
containerNodeURI=localhost.localdomain:8042, containerResourceMemory=1024
2014-06-16 07:10:13,333 INFO  [pool-2-thread-1]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - Setting
up container launch container for
containerid=container_1402830191668_0017_01_000002
2014-06-16 07:10:13,348 INFO  [pool-2-thread-1]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - Conatain
launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH}
org.apache.giraph.yarn.GiraphYarnTask 1402830191668 17 2 1
1><LOG_DIR>/task-2-stdout.log 2><LOG_DIR>/task-2-stderr.log
2014-06-16 07:10:13,349 INFO  [pool-2-thread-1]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - Setting
username in ContainerLaunchContext to: hadoop
2014-06-16 07:10:13,744 INFO  [pool-2-thread-1] yarn.YarnUtils
(YarnUtils.java:addFsResourcesToMap(72)) - Adding
giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
to LocalResources for export.to
hdfs://localhost.localdomain:8020/user/hadoop/giraph_yarn_jar_cache/application_1402830191668_0017/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
2014-06-16 07:10:13,774 INFO  [pool-2-thread-1] yarn.YarnUtils
(YarnUtils.java:addFileToResourceMap(160)) - Registered file in
LocalResources ::
hdfs://localhost.localdomain:8020/user/hadoop/giraph_yarn_jar_cache/application_1402830191668_0017/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
2014-06-16 07:10:13,774 WARN  [pool-2-thread-1] yarn.YarnUtils
(YarnUtils.java:addFsResourcesToMap(81)) - Job jars (-yj option) didn't
include giraph-core.
2014-06-16 07:10:13,776 INFO  [pool-2-thread-1] yarn.YarnUtils
(YarnUtils.java:addFileToResourceMap(160)) - Registered file in
LocalResources ::
hdfs://localhost.localdomain:8020/user/hadoop/giraph_yarn_jar_cache/application_1402830191668_0017/giraph-conf.xml
2014-06-16 07:10:13,786 INFO
 [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing Event
EventType: START_CONTAINER for Container
container_1402830191668_0017_01_000002
2014-06-16 07:10:13,787 INFO
 [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
impl.ContainerManagementProtocolProxy
(ContainerManagementProtocolProxy.java:newProxy(205)) - Opening proxy :
localhost.localdomain:38256
2014-06-16 07:10:13,829 INFO
 [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1]
impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing Event
EventType: QUERY_CONTAINER for Container
container_1402830191668_0017_01_000002
2014-06-16 07:10:14,337 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersAllocated(605)) - Got response
from RM for container ask, allocatedCnt=1
2014-06-16 07:10:14,337 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersAllocated(608)) - Total allocated
# of container so far : 2 allocated out of 2 required.
2014-06-16 07:10:14,337 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) -
Launching command on a new container.,
containerId=container_1402830191668_0017_01_000003,
containerNode=localhost.localdomain:38256,
containerNodeURI=localhost.localdomain:8042, containerResourceMemory=1024
2014-06-16 07:10:14,339 INFO  [pool-2-thread-2]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - Setting
up container launch container for
containerid=container_1402830191668_0017_01_000003
2014-06-16 07:10:14,339 INFO  [pool-2-thread-2]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - Conatain
launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH}
org.apache.giraph.yarn.GiraphYarnTask 1402830191668 17 3 1
1><LOG_DIR>/task-3-stdout.log 2><LOG_DIR>/task-3-stderr.log
2014-06-16 07:10:14,339 INFO  [pool-2-thread-2]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - Setting
username in ContainerLaunchContext to: hadoop
2014-06-16 07:10:14,342 INFO
 [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #2]
impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing Event
EventType: START_CONTAINER for Container
container_1402830191668_0017_01_000003
2014-06-16 07:10:14,348 INFO
 [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #3]
impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing Event
EventType: QUERY_CONTAINER for Container
container_1402830191668_0017_01_000003
2014-06-16 07:10:15,341 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersCompleted(571)) - Got response
from RM for container ask, completedCnt=2
2014-06-16 07:10:15,342 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersCompleted(574)) - Got container
status for containerID=container_1402830191668_0017_01_000002,
state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)



2014-06-16 07:10:15,342 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersCompleted(574)) - Got container
status for containerID=container_1402830191668_0017_01_000003,
state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)



2014-06-16 07:10:15,343 INFO  [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersCompleted(594)) - All container
compeleted. done = true
2014-06-16 07:10:15,517 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:run(194)) - Done true
2014-06-16 07:10:15,518 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:run(198)) - Forcefully terminating executors
with done =:true
2014-06-16 07:10:15,518 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:finish(212)) - Application completed.
Stopping running containers
2014-06-16 07:10:15,542 INFO  [main] impl.ContainerManagementProtocolProxy
(ContainerManagementProtocolProxy.java:mayBeCloseProxy(138)) - Closing
proxy : localhost.localdomain:38256
2014-06-16 07:10:15,542 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:finish(217)) - Application completed.
Signalling finish to RM
2014-06-16 07:10:15,554 INFO  [main] impl.AMRMClientImpl
(AMRMClientImpl.java:unregisterApplicationMaster(316)) - Waiting for
application to be successfully unregistered.
2014-06-16 07:10:15,656 INFO  [AMRM Heartbeater thread]
impl.AMRMClientAsyncImpl (AMRMClientAsyncImpl.java:run(251)) - Heartbeater
interrupted
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:249)
2014-06-16 07:10:15,658 INFO  [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:main(445)) - Giraph Application Master
failed. exiting
2014-06-16 07:10:15,659 INFO  [AMRM Callback Handler Thread]
impl.AMRMClientAsyncImpl (AMRMClientAsyncImpl.java:run(277)) - Interrupted
while waiting for queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)

Any ideas? :)

Thanks

--John