You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Phillip Rhodes <mo...@gmail.com> on 2015/03/10 04:39:44 UTC
Trouble running Giraph on YARN / Hadoop 2.5.2
Folks:
I'm trying to run the basic ShortestPaths example using Giraph 1.1 on
Hadoop 2.5.2. I'm running in actual cluster model (eg, not
psuedo-distributed) and I can run standard mapreduce jobs OK. But
when I try to run the Giraph example, it seems to hang unless I set
-ca giraph.SplitMasterWorker=false
and correspondingly set number of workers to 1. But this kinda
defeats the point of running on a cluster, no? OTOH, if I run
without disabling SplitMasterWorker, I get this error:
When using LocalJobRunner, you cannot run in split master / worker mode
since there is only 1 task at a time!
which suggests that Girpah is defaulting to local mode. One report I
read suggested fixing this by adding
-ca mapred.job.tracker=10.0.0.12:5431
to the Girpah command line, but on Hadoop 2.5.2 with YARN, there is no
JobTracker on port 5431, if I understand correctly. Anyway, if I do
add that bit, the job **tries** to run, but seems to hang without ever
finishing. Here's the complete command line, and the job output
follows:
[prhodes@ip-10-0-0-12 conf]$ hadoop jar /home/prhodes/giraph/giraph-
examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.5.2-jar-with-
dependencies.jar org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /user/prhodes/input/tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/prhodes/giraph_output/shortestpaths -w 3 -ca
mapred.job.tracker=10.0.0.12:5431
15/03/10 03:18:59 INFO utils.ConfigurationUtils: No edge input
format specified. Ensure your InputFormat does not require one.
15/03/10 03:19:02 INFO server.NIOServerCnxnFactory: binding to
port 0.0.0.0/0.0.0.0:22181
15/03/10 03:19:02 INFO server.PrepRequestProcessor:
zookeeper.skipACL=="yes", ACL checks will be skipped
15/03/10 03:19:05 INFO zk.ZooKeeperManager:
onlineZooKeeperServers: Connect attempt 1 of 10 max trying to connect
to ip-10-0-0-12.ec2.internal:22181 with poll msecs = 3000
15/03/10 03:19:05 INFO zk.ZooKeeperManager:
onlineZooKeeperServers: Connected to
ip-10-0-0-12.ec2.internal/10.0.0.12:22181!
15/03/10 03:19:05 INFO zk.ZooKeeperManager:
onlineZooKeeperServers: Creating my filestamp
_bsp/_defaultZkManagerDir/job_local1346154675_0001/_zkServer/ip-10-0-0-12.ec2.internal
0
15/03/10 03:19:05 INFO server.NIOServerCnxnFactory: Accepted
socket connection from /10.0.0.12:45182
15/03/10 03:19:05 INFO graph.GraphTaskManager: setup: Chosen to
run ZooKeeper...
15/03/10 03:19:05 INFO graph.GraphTaskManager: setup: Starting up
BspServiceMaster (master thread)...
15/03/10 03:19:05 INFO bsp.BspService: BspService: Path to create
to halt is /_hadoopBsp/job_local1346154675_0001/_haltComputation
15/03/10 03:19:05 INFO bsp.BspService: BspService: Connecting to
ZooKeeper with job job_local1346154675_0001, 0 on
ip-10-0-0-12.ec2.internal:22181
15/03/10 03:19:05 INFO zookeeper.ClientCnxn: Opening socket
connection to server ip-10-0-0-12.ec2.internal/10.0.0.12:22181. Will
not attempt to authenticate using SASL (unknown error)
15/03/10 03:19:05 INFO server.NIOServerCnxnFactory: Accepted
socket connection from /10.0.0.12:45183
15/03/10 03:19:05 INFO zookeeper.ClientCnxn: Socket connection
established to ip-10-0-0-12.ec2.internal/10.0.0.12:22181, initiating
session
15/03/10 03:19:05 INFO server.ZooKeeperServer: Client attempting
to establish new session at /10.0.0.12:45183
15/03/10 03:19:05 INFO persistence.FileTxnLog: Creating new log file: log.1
15/03/10 03:19:05 INFO server.ZooKeeperServer: Established session
0x14c01b158f00000 with negotiated timeout 600000 for client
/10.0.0.12:45183
15/03/10 03:19:05 INFO zookeeper.ClientCnxn: Session establishment
complete on server ip-10-0-0-12.ec2.internal/10.0.0.12:22181,
sessionid = 0x14c01b158f00000, negotiated timeout = 600000
15/03/10 03:19:05 INFO bsp.BspService: process: Asynchronous
connection complete.
15/03/10 03:19:05 INFO graph.GraphTaskManager: map: No need to do
anything when not a worker
15/03/10 03:19:05 INFO graph.GraphTaskManager: cleanup: Starting
for MASTER_ZOOKEEPER_ONLY
15/03/10 03:19:05 INFO server.PrepRequestProcessor: Got user-level
KeeperException when processing sessionid:0x14c01b158f00000
type:create cxid:0x1 zxid:0x2 txntype:-1 reqpath:n/a Error
Path:/_hadoopBsp/job_local1346154675_0001/_masterElectionDir
Error:KeeperErrorCode = NoNode for
/_hadoopBsp/job_local1346154675_0001/_masterElectionDir
15/03/10 03:19:05 INFO master.BspServiceMaster: becomeMaster:
First child is '/_hadoopBsp/job_local1346154675_0001/_masterElectionDir/ip-10-0-0-12.ec2.internal_00000000000'
and my bid is '/_hadoopBsp/job_local1346154675_0001/_masterElectionDir/ip-10-0-0-12.ec2.internal_00000000000'
15/03/10 03:19:05 INFO netty.NettyServer: NettyServer: Using
execution group with 8 threads for requestFrameDecoder.
15/03/10 03:19:05 INFO Configuration.deprecation: mapred.map.tasks
is deprecated. Instead, use mapreduce.job.maps
15/03/10 03:19:05 INFO netty.NettyServer: start: Started server
communication server: ip-10-0-0-12.ec2.internal/10.0.0.12:30000 with
up to 16 threads on bind attempt 0 with sendBufferSize = 32768
receiveBufferSize = 524288
15/03/10 03:19:05 INFO netty.NettyClient: NettyClient: Using
execution handler with 8 threads after request-encoder.
15/03/10 03:19:05 INFO master.BspServiceMaster: becomeMaster: I am
now the master!
15/03/10 03:19:05 INFO server.PrepRequestProcessor: Got user-level
KeeperException when processing sessionid:0x14c01b158f00000
type:create cxid:0xe zxid:0x9 txntype:-1 reqpath:n/a Error
Path:/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0
Error:KeeperErrorCode = NoNode for
/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0
15/03/10 03:19:05 INFO bsp.BspService: process:
applicationAttemptChanged signaled
15/03/10 03:19:05 INFO server.PrepRequestProcessor: Got user-level
KeeperException when processing sessionid:0x14c01b158f00000
type:create cxid:0x16 zxid:0xc txntype:-1 reqpath:n/a Error
Path:/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir/-1
Error:KeeperErrorCode = NoNode for
/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir/-1
15/03/10 03:19:05 WARN bsp.BspService: process: Unknown and
unprocessed event
(path=/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir,
type=NodeChildrenChanged, state=SyncConnected)
15/03/10 03:19:07 INFO mapred.LocalJobRunner:
MASTER_ZOOKEEPER_ONLY checkWorkers: Only found 0 responses of 3 needed
to start superstep -1 > map
15/03/10 03:19:07 INFO
job.HaltApplicationUtils$DefaultHaltInstructionsWriter:
writeHaltInstructions: To halt after next superstep execute:
'bin/halt-application --zkServer ip-10-0-0-12.ec2.internal:22181
--zkNode /_hadoopBsp/job_local1346154675_0001/_haltComputation'
15/03/10 03:19:07 INFO mapreduce.Job: Running job: job_local1346154675_0001
15/03/10 03:19:08 INFO mapreduce.Job: Job job_local1346154675_0001
running in uber mode : false
15/03/10 03:19:08 INFO mapreduce.Job: map 25% reduce 0%
15/03/10 03:19:10 INFO mapred.LocalJobRunner:
MASTER_ZOOKEEPER_ONLY checkWorkers: Only found 0 responses of 3 needed
to start superstep -1 > map
15/03/10 03:19:19 INFO mapred.LocalJobRunner:
MASTER_ZOOKEEPER_ONLY checkWorkers: Only found 0 responses of 3 needed
to start superstep -1 > map
15/03/10 03:19:28 INFO mapred.LocalJobRunner:
MASTER_ZOOKEEPER_ONLY checkWorkers: Only found 0 responses of 3 needed
to start superstep -1 > map
15/03/10 03:19:35 INFO master.BspServiceMaster: checkWorkers: Only
found 0 responses of 3 needed to start superstep -1. Reporting every
30000 msecs, 569976 more msecs left before giving up.
15/03/10 03:19:35 INFO server.PrepRequestProcessor: Got user-level
KeeperException when processing sessionid:0x14c01b158f00000
type:create cxid:0x22 zxid:0x10 txntype:-1 reqpath:n/a Error
Path:/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir/-1/_workerHealthyDir
Error:KeeperErrorCode = NodeExists for
/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir/-1/_workerHealthyDir
15/03/10 03:19:35 INFO server.PrepRequestProcessor: Got user-level
KeeperException when processing sessionid:0x14c01b158f00000
type:create cxid:0x23 zxid:0x11 txntype:-1 reqpath:n/a Error
Path:/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir/-1/_workerUnhealthyDir
Error:KeeperErrorCode = NodeExists for
/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir/-1/_workerUnhealthyDir
15/03/10 03:19:40 INFO mapred.LocalJobRunner:
MASTER_ZOOKEEPER_ONLY checkWorkers: Only found 0 responses of 3 needed
to start superstep -1 > map
Any help is much appreciated!
Phil
Re: Trouble running Giraph on YARN / Hadoop 2.5.2
Posted by Phillip Rhodes <mo...@gmail.com>.
OK, got by this problem. It turns out, I built Giraph using the
hadoop_2 profile, and not hadoop_yarn. When I switched to using mvn
-Phadoop_yarn, the problem I was experiencing changed... I now see log
messages showing Girpah connecting to the Yarn ResourceManager on port
8032, which is good. Unfortunately I now get a *different* error
related to the heap size, but I'll post that in a separate message.
Phil
This message optimized for indexing by NSA PRISM
On Mon, Mar 9, 2015 at 11:39 PM, Phillip Rhodes
<mo...@gmail.com> wrote:
> Folks:
>
> I'm trying to run the basic ShortestPaths example using Giraph 1.1 on
> Hadoop 2.5.2. I'm running in actual cluster model (eg, not
> psuedo-distributed) and I can run standard mapreduce jobs OK. But
> when I try to run the Giraph example, it seems to hang unless I set
>
> -ca giraph.SplitMasterWorker=false
>
> and correspondingly set number of workers to 1. But this kinda
> defeats the point of running on a cluster, no? OTOH, if I run
> without disabling SplitMasterWorker, I get this error:
>
> When using LocalJobRunner, you cannot run in split master / worker mode
> since there is only 1 task at a time!
>
> which suggests that Girpah is defaulting to local mode. One report I
> read suggested fixing this by adding
>
> -ca mapred.job.tracker=10.0.0.12:5431
>
> to the Girpah command line, but on Hadoop 2.5.2 with YARN, there is no
> JobTracker on port 5431, if I understand correctly. Anyway, if I do
> add that bit, the job **tries** to run, but seems to hang without ever
> finishing. Here's the complete command line, and the job output
> follows:
>
>
> [prhodes@ip-10-0-0-12 conf]$ hadoop jar /home/prhodes/giraph/giraph-
> examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.5.2-jar-with-
> dependencies.jar org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip /user/prhodes/input/tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /user/prhodes/giraph_output/shortestpaths -w 3 -ca
> mapred.job.tracker=10.0.0.12:5431
>
>
>
>
> 15/03/10 03:18:59 INFO utils.ConfigurationUtils: No edge input
> format specified. Ensure your InputFormat does not require one.
> 15/03/10 03:19:02 INFO server.NIOServerCnxnFactory: binding to
> port 0.0.0.0/0.0.0.0:22181
> 15/03/10 03:19:02 INFO server.PrepRequestProcessor:
> zookeeper.skipACL=="yes", ACL checks will be skipped
> 15/03/10 03:19:05 INFO zk.ZooKeeperManager:
> onlineZooKeeperServers: Connect attempt 1 of 10 max trying to connect
> to ip-10-0-0-12.ec2.internal:22181 with poll msecs = 3000
> 15/03/10 03:19:05 INFO zk.ZooKeeperManager:
> onlineZooKeeperServers: Connected to
> ip-10-0-0-12.ec2.internal/10.0.0.12:22181!
> 15/03/10 03:19:05 INFO zk.ZooKeeperManager:
> onlineZooKeeperServers: Creating my filestamp
> _bsp/_defaultZkManagerDir/job_local1346154675_0001/_zkServer/ip-10-0-0-12.ec2.internal
> 0
> 15/03/10 03:19:05 INFO server.NIOServerCnxnFactory: Accepted
> socket connection from /10.0.0.12:45182
> 15/03/10 03:19:05 INFO graph.GraphTaskManager: setup: Chosen to
> run ZooKeeper...
> 15/03/10 03:19:05 INFO graph.GraphTaskManager: setup: Starting up
> BspServiceMaster (master thread)...
> 15/03/10 03:19:05 INFO bsp.BspService: BspService: Path to create
> to halt is /_hadoopBsp/job_local1346154675_0001/_haltComputation
> 15/03/10 03:19:05 INFO bsp.BspService: BspService: Connecting to
> ZooKeeper with job job_local1346154675_0001, 0 on
> ip-10-0-0-12.ec2.internal:22181
> 15/03/10 03:19:05 INFO zookeeper.ClientCnxn: Opening socket
> connection to server ip-10-0-0-12.ec2.internal/10.0.0.12:22181. Will
> not attempt to authenticate using SASL (unknown error)
> 15/03/10 03:19:05 INFO server.NIOServerCnxnFactory: Accepted
> socket connection from /10.0.0.12:45183
> 15/03/10 03:19:05 INFO zookeeper.ClientCnxn: Socket connection
> established to ip-10-0-0-12.ec2.internal/10.0.0.12:22181, initiating
> session
> 15/03/10 03:19:05 INFO server.ZooKeeperServer: Client attempting
> to establish new session at /10.0.0.12:45183
> 15/03/10 03:19:05 INFO persistence.FileTxnLog: Creating new log file: log.1
> 15/03/10 03:19:05 INFO server.ZooKeeperServer: Established session
> 0x14c01b158f00000 with negotiated timeout 600000 for client
> /10.0.0.12:45183
> 15/03/10 03:19:05 INFO zookeeper.ClientCnxn: Session establishment
> complete on server ip-10-0-0-12.ec2.internal/10.0.0.12:22181,
> sessionid = 0x14c01b158f00000, negotiated timeout = 600000
> 15/03/10 03:19:05 INFO bsp.BspService: process: Asynchronous
> connection complete.
> 15/03/10 03:19:05 INFO graph.GraphTaskManager: map: No need to do
> anything when not a worker
> 15/03/10 03:19:05 INFO graph.GraphTaskManager: cleanup: Starting
> for MASTER_ZOOKEEPER_ONLY
> 15/03/10 03:19:05 INFO server.PrepRequestProcessor: Got user-level
> KeeperException when processing sessionid:0x14c01b158f00000
> type:create cxid:0x1 zxid:0x2 txntype:-1 reqpath:n/a Error
> Path:/_hadoopBsp/job_local1346154675_0001/_masterElectionDir
> Error:KeeperErrorCode = NoNode for
> /_hadoopBsp/job_local1346154675_0001/_masterElectionDir
> 15/03/10 03:19:05 INFO master.BspServiceMaster: becomeMaster:
> First child is '/_hadoopBsp/job_local1346154675_0001/_masterElectionDir/ip-10-0-0-12.ec2.internal_00000000000'
> and my bid is '/_hadoopBsp/job_local1346154675_0001/_masterElectionDir/ip-10-0-0-12.ec2.internal_00000000000'
> 15/03/10 03:19:05 INFO netty.NettyServer: NettyServer: Using
> execution group with 8 threads for requestFrameDecoder.
> 15/03/10 03:19:05 INFO Configuration.deprecation: mapred.map.tasks
> is deprecated. Instead, use mapreduce.job.maps
> 15/03/10 03:19:05 INFO netty.NettyServer: start: Started server
> communication server: ip-10-0-0-12.ec2.internal/10.0.0.12:30000 with
> up to 16 threads on bind attempt 0 with sendBufferSize = 32768
> receiveBufferSize = 524288
> 15/03/10 03:19:05 INFO netty.NettyClient: NettyClient: Using
> execution handler with 8 threads after request-encoder.
> 15/03/10 03:19:05 INFO master.BspServiceMaster: becomeMaster: I am
> now the master!
> 15/03/10 03:19:05 INFO server.PrepRequestProcessor: Got user-level
> KeeperException when processing sessionid:0x14c01b158f00000
> type:create cxid:0xe zxid:0x9 txntype:-1 reqpath:n/a Error
> Path:/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0
> Error:KeeperErrorCode = NoNode for
> /_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0
> 15/03/10 03:19:05 INFO bsp.BspService: process:
> applicationAttemptChanged signaled
> 15/03/10 03:19:05 INFO server.PrepRequestProcessor: Got user-level
> KeeperException when processing sessionid:0x14c01b158f00000
> type:create cxid:0x16 zxid:0xc txntype:-1 reqpath:n/a Error
> Path:/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir/-1
> Error:KeeperErrorCode = NoNode for
> /_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir/-1
> 15/03/10 03:19:05 WARN bsp.BspService: process: Unknown and
> unprocessed event
> (path=/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir,
> type=NodeChildrenChanged, state=SyncConnected)
> 15/03/10 03:19:07 INFO mapred.LocalJobRunner:
> MASTER_ZOOKEEPER_ONLY checkWorkers: Only found 0 responses of 3 needed
> to start superstep -1 > map
> 15/03/10 03:19:07 INFO
> job.HaltApplicationUtils$DefaultHaltInstructionsWriter:
> writeHaltInstructions: To halt after next superstep execute:
> 'bin/halt-application --zkServer ip-10-0-0-12.ec2.internal:22181
> --zkNode /_hadoopBsp/job_local1346154675_0001/_haltComputation'
> 15/03/10 03:19:07 INFO mapreduce.Job: Running job: job_local1346154675_0001
> 15/03/10 03:19:08 INFO mapreduce.Job: Job job_local1346154675_0001
> running in uber mode : false
> 15/03/10 03:19:08 INFO mapreduce.Job: map 25% reduce 0%
> 15/03/10 03:19:10 INFO mapred.LocalJobRunner:
> MASTER_ZOOKEEPER_ONLY checkWorkers: Only found 0 responses of 3 needed
> to start superstep -1 > map
> 15/03/10 03:19:19 INFO mapred.LocalJobRunner:
> MASTER_ZOOKEEPER_ONLY checkWorkers: Only found 0 responses of 3 needed
> to start superstep -1 > map
> 15/03/10 03:19:28 INFO mapred.LocalJobRunner:
> MASTER_ZOOKEEPER_ONLY checkWorkers: Only found 0 responses of 3 needed
> to start superstep -1 > map
> 15/03/10 03:19:35 INFO master.BspServiceMaster: checkWorkers: Only
> found 0 responses of 3 needed to start superstep -1. Reporting every
> 30000 msecs, 569976 more msecs left before giving up.
> 15/03/10 03:19:35 INFO server.PrepRequestProcessor: Got user-level
> KeeperException when processing sessionid:0x14c01b158f00000
> type:create cxid:0x22 zxid:0x10 txntype:-1 reqpath:n/a Error
> Path:/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir/-1/_workerHealthyDir
> Error:KeeperErrorCode = NodeExists for
> /_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir/-1/_workerHealthyDir
> 15/03/10 03:19:35 INFO server.PrepRequestProcessor: Got user-level
> KeeperException when processing sessionid:0x14c01b158f00000
> type:create cxid:0x23 zxid:0x11 txntype:-1 reqpath:n/a Error
> Path:/_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir/-1/_workerUnhealthyDir
> Error:KeeperErrorCode = NodeExists for
> /_hadoopBsp/job_local1346154675_0001/_applicationAttemptsDir/0/_superstepDir/-1/_workerUnhealthyDir
> 15/03/10 03:19:40 INFO mapred.LocalJobRunner:
> MASTER_ZOOKEEPER_ONLY checkWorkers: Only found 0 responses of 3 needed
> to start superstep -1 > map
>
> Any help is much appreciated!
>
>
> Phil