You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Simon McGloin <si...@gmail.com> on 2013/10/17 19:39:08 UTC
Master always fails on dataset
Hey Guys.
I have a problem running my giraph job on a dataset with 20,000,000 edges
and 2,000,000 vertices. All the vertices are Text based. The giraph job
works perfectly on smaller datasets but always fails on larger ones. The
setup I have is a 3 node cluster, each with 24 cores and 24 GB of ram. The
cluster has a total of 60 mappers each with mapred.child.java.opts set to
-Xmx1000m.
If I don't use the Out-of-Core option then the job fails due to running out
of java heap space. When I use -Dgiraph.useOutOfCoreGraph=true then the
master eventually fails due to a worker disconnecting from zookeeper. The
worker just throws a warning and doesn't actually fail. I've been using the
-Dgiraph.checkpointFrequency=1 option but this doesn't seem to restart the
mapper. I'm new to zookeeper too so if this is a zookeeper problem then let
me know and I can investigate it as such.
Below is the options I'm using and the errors I'm currently getting
Any help or tips are appreciated,
Simon
Options:
-Dgiraph.zkList=10.10.5.103:2181,10.10.5.104:2181,10.10.5.105:2181
-Dgiraph.checkpointFrequency=1
-Dgiraph.useOutOfCoreGraph=true
-Dgiraph.zkSessionMsecTimeout=600000
-Dgiraph.numComputeThreads=2
Master Log:
2013-10-17 18:19:34,638 INFO org.apache.giraph.master.BspServiceMaster:
barrierOnWorkerList: 0 out of 50 workers finished on superstep 1 on path
/_hadoopBsp/job_201310161506_0064/_applicationAttemptsDir/0/_superstepDir/1/_workerWroteCheckpointDir
2013-10-17 18:20:52,105 ERROR org.apache.giraph.master.BspServiceMaster:
superstepChosenWorkerAlive: Missing chosen worker Worker(hostname=
node1.mycompany.com, MRtaskID=30, port=30030) on superstep 1
2013-10-17 18:20:52,106 INFO org.apache.giraph.master.MasterThread:
masterThread: Coordination of superstep 1 took 78.851 seconds ended with
state WORKER_FAILURE and is now on superstep 1
2013-10-17 18:20:52,112 ERROR org.apache.giraph.master.MasterThread:
masterThread: Master algorithm failed with RuntimeException
java.lang.RuntimeException: restartFromCheckpoint: KeeperException
at
org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1185)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:135)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/_hadoopBsp/job_201310161506_0064/_vertexInputSplitDir
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:728)
at org.apache.giraph.zk.ZooKeeperExt.deleteExt(ZooKeeperExt.java:307)
at
org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1177)
... 1 more
2013-10-17 18:20:52,115 FATAL org.apache.giraph.graph.GraphMapper:
uncaughtException: OverrideExceptionHandler on thread
org.apache.giraph.master.MasterThread, msg = java.lang.RuntimeException:
restartFromCheckpoint: KeeperException, exiting...
java.lang.IllegalStateException: java.lang.RuntimeException:
restartFromCheckpoint: KeeperException
at org.apache.giraph.master.MasterThread.run(MasterThread.java:181)
Caused by: java.lang.RuntimeException: restartFromCheckpoint:
KeeperException
at
org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1185)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:135)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/_hadoopBsp/job_201310161506_0064/_vertexInputSplitDir
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:728)
at org.apache.giraph.zk.ZooKeeperExt.deleteExt(ZooKeeperExt.java:307)
at
org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1177)
... 1 more
Worker 30 log:
2013-10-17 18:19:07,309 INFO
org.apache.giraph.partition.DiskBackedPartitionStore: offloadPartition:
writing partition edges 1927 to
/data/var/hdfs/data/mapred/taskTracker/simon/jobcache/job_201310161506_0064/attempt_201310161506_0064_m_000030_0/work/_bsp/_partitions/job_201310161506_0064/partition-1927_edges
2013-10-17 18:19:45,736 INFO org.apache.giraph.utils.ProgressableUtils:
waitFor: Future result not ready yet java.util.concurrent.FutureTask@c07bacb
2013-10-17 18:19:45,737 INFO org.apache.giraph.utils.ProgressableUtils:
waitFor: Waiting for
org.apache.giraph.utils.ProgressableUtils$FutureWaitable@4f786b98
2013-10-17 18:19:45,789 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 40183ms for sessionid
0x341c716ad860073, closing socket connection and attempting reconnect
2013-10-17 18:19:46,113 WARN org.apache.giraph.bsp.BspService: process:
Disconnected from ZooKeeper (will automatically try to recover)
WatchedEvent state:Disconnected type:None path:null
2013-10-17 18:19:46,113 WARN org.apache.giraph.worker.InputSplitsHandler:
process: Problem with zookeeper, got event with path null, state
Disconnected, event type None
2013-10-17 18:19:46,746 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server /10.10.5.105:2181
2013-10-17 18:19:46,747 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to node3.mycompany.com/10.10.5.105:2181, initiating
session
2013-10-17 18:19:46,750 INFO org.apache.zookeeper.ClientCnxn: Unable to
reconnect to ZooKeeper service, session 0x341c716ad860073 has expired,
closing socket connection
2013-10-17 18:19:46,750 WARN org.apache.giraph.bsp.BspService: process: Got
unknown null path event WatchedEvent state:Expired type:None path:null
2013-10-17 18:19:46,750 WARN org.apache.giraph.worker.InputSplitsHandler:
process: Problem with zookeeper, got event with path null, state Expired,
event type None
2013-10-17 18:19:46,750 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2013-10-17 18:20:33,546 INFO
org.apache.giraph.comm.netty.handler.RequestDecoder: decode: Server window
metrics MBytes/sec sent = 0, MBytes/sec received = 0.0059, MBytesSent =
0.0008, MBytesReceived = 0.7636, ave sent req MBytes = 0, ave received req
MBytes = 0.0111, secs waited = 128.396
2013-10-17 18:20:45,737 INFO org.apache.giraph.utils.ProgressableUtils:
waitFor: Future result not ready yet java.util.concurrent.FutureTask@c07bacb
2013-10-17 18:20:45,737 INFO org.apache.giraph.utils.ProgressableUtils:
waitFor: Waiting for
org.apache.giraph.utils.ProgressableUtils$FutureWaitable@4f786b98
RE: Master always fails on dataset
Posted by "Tunvall, Fredrik" <Fr...@ovum.com>.
Please disregard - Outlook sent it to the wrong address. Sorry. - F
From: Tunvall, Fredrik [mailto:Fredrik.Tunvall@ovum.com]
Sent: Friday, October 18, 2013 12:25 PM
To: user@giraph.apache.org
Subject: RE: Master always fails on dataset
I will reach out right now
From: Simon McGloin [mailto:simonmcgloin@gmail.com]<mailto:[mailto:simonmcgloin@gmail.com]>
Sent: Friday, October 18, 2013 12:24 PM
To: user@giraph.apache.org<ma...@giraph.apache.org>
Subject: Re: Master always fails on dataset
Thanks Claudio. Yes the machines are homogenous. Unfortunately I don't have ganglia installed. You were right it is a memory issue. I've reduced the number of partitions down to 1 with -Dgiraph.maxPartitionsInMemory=1 and now my jobs are failing due to running out of diskspace on HDFS. Each HDFS mount has 100gb of space. I will increase the size of HDFS and order more memory next week. Is there anyway to calculate the memory requirements of a giraph job? I presume it depends on the algorithm being run.
On Thu, Oct 17, 2013 at 6:42 PM, Claudio Martella <cl...@gmail.com>> wrote:
Try decreasing the number of partitions you keep in memory. You're running out of memory. Also, are your nodes homogenous? It could be one particular machine swapping or something. If you have ganglia, try investigating the usage of memory.
On Thu, Oct 17, 2013 at 7:39 PM, Simon McGloin <si...@gmail.com>> wrote:
Hey Guys.
I have a problem running my giraph job on a dataset with 20,000,000 edges and 2,000,000 vertices. All the vertices are Text based. The giraph job works perfectly on smaller datasets but always fails on larger ones. The setup I have is a 3 node cluster, each with 24 cores and 24 GB of ram. The cluster has a total of 60 mappers each with mapred.child.java.opts set to -Xmx1000m.
If I don't use the Out-of-Core option then the job fails due to running out of java heap space. When I use -Dgiraph.useOutOfCoreGraph=true then the master eventually fails due to a worker disconnecting from zookeeper. The worker just throws a warning and doesn't actually fail. I've been using the -Dgiraph.checkpointFrequency=1 option but this doesn't seem to restart the mapper. I'm new to zookeeper too so if this is a zookeeper problem then let me know and I can investigate it as such.
Below is the options I'm using and the errors I'm currently getting
Any help or tips are appreciated,
Simon
Options:
-Dgiraph.zkList=10.10.5.103:2181<http://10.10.5.103:2181>,10.10.5.104:2181<http://10.10.5.104:2181>,10.10.5.105:2181<http://10.10.5.105:2181>
-Dgiraph.checkpointFrequency=1
-Dgiraph.useOutOfCoreGraph=true
-Dgiraph.zkSessionMsecTimeout=600000
-Dgiraph.numComputeThreads=2
Master Log:
2013-10-17 18:19:34,638 INFO org.apache.giraph.master.BspServiceMaster: barrierOnWorkerList: 0 out of 50 workers finished on superstep 1 on path /_hadoopBsp/job_201310161506_0064/_applicationAttemptsDir/0/_superstepDir/1/_workerWroteCheckpointDir
2013-10-17 18:20:52,105 ERROR org.apache.giraph.master.BspServiceMaster: superstepChosenWorkerAlive: Missing chosen worker Worker(hostname=node1.mycompany.com<http://node1.mycompany.com>, MRtaskID=30, port=30030) on superstep 1
2013-10-17 18:20:52,106 INFO org.apache.giraph.master.MasterThread: masterThread: Coordination of superstep 1 took 78.851 seconds ended with state WORKER_FAILURE and is now on superstep 1
2013-10-17 18:20:52,112 ERROR org.apache.giraph.master.MasterThread: masterThread: Master algorithm failed with RuntimeException
java.lang.RuntimeException: restartFromCheckpoint: KeeperException
at org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1185)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:135)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /_hadoopBsp/job_201310161506_0064/_vertexInputSplitDir
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:728)
at org.apache.giraph.zk.ZooKeeperExt.deleteExt(ZooKeeperExt.java:307)
at org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1177)
... 1 more
2013-10-17 18:20:52,115 FATAL org.apache.giraph.graph.GraphMapper: uncaughtException: OverrideExceptionHandler on thread org.apache.giraph.master.MasterThread, msg = java.lang.RuntimeException: restartFromCheckpoint: KeeperException, exiting...
java.lang.IllegalStateException: java.lang.RuntimeException: restartFromCheckpoint: KeeperException
at org.apache.giraph.master.MasterThread.run(MasterThread.java:181)
Caused by: java.lang.RuntimeException: restartFromCheckpoint: KeeperException
at org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1185)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:135)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /_hadoopBsp/job_201310161506_0064/_vertexInputSplitDir
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:728)
at org.apache.giraph.zk.ZooKeeperExt.deleteExt(ZooKeeperExt.java:307)
at org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1177)
... 1 more
Worker 30 log:
2013-10-17 18:19:07,309 INFO org.apache.giraph.partition.DiskBackedPartitionStore: offloadPartition: writing partition edges 1927 to /data/var/hdfs/data/mapred/taskTracker/simon/jobcache/job_201310161506_0064/attempt_201310161506_0064_m_000030_0/work/_bsp/_partitions/job_201310161506_0064/partition-1927_edges
2013-10-17 18:19:45,736 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Future result not ready yet java.util.concurrent.FutureTask@c07bacb<ma...@c07bacb>
2013-10-17 18:19:45,737 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@4f786b98<ma...@4f786b98>
2013-10-17 18:19:45,789 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 40183ms for sessionid 0x341c716ad860073, closing socket connection and attempting reconnect
2013-10-17 18:19:46,113 WARN org.apache.giraph.bsp.BspService: process: Disconnected from ZooKeeper (will automatically try to recover) WatchedEvent state:Disconnected type:None path:null
2013-10-17 18:19:46,113 WARN org.apache.giraph.worker.InputSplitsHandler: process: Problem with zookeeper, got event with path null, state Disconnected, event type None
2013-10-17 18:19:46,746 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.10.5.105:2181<http://10.10.5.105:2181>
2013-10-17 18:19:46,747 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to node3.mycompany.com/10.10.5.105:2181<http://node3.mycompany.com/10.10.5.105:2181>, initiating session
2013-10-17 18:19:46,750 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x341c716ad860073 has expired, closing socket connection
2013-10-17 18:19:46,750 WARN org.apache.giraph.bsp.BspService: process: Got unknown null path event WatchedEvent state:Expired type:None path:null
2013-10-17 18:19:46,750 WARN org.apache.giraph.worker.InputSplitsHandler: process: Problem with zookeeper, got event with path null, state Expired, event type None
2013-10-17 18:19:46,750 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2013-10-17 18:20:33,546 INFO org.apache.giraph.comm.netty.handler.RequestDecoder: decode: Server window metrics MBytes/sec sent = 0, MBytes/sec received = 0.0059, MBytesSent = 0.0008, MBytesReceived = 0.7636, ave sent req MBytes = 0, ave received req MBytes = 0.0111, secs waited = 128.396
2013-10-17 18:20:45,737 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Future result not ready yet java.util.concurrent.FutureTask@c07bacb<ma...@c07bacb>
2013-10-17 18:20:45,737 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@4f786b98<ma...@4f786b98>
--
Claudio Martella
claudio.martella@gmail.com<ma...@gmail.com>
RE: Master always fails on dataset
Posted by "Tunvall, Fredrik" <Fr...@ovum.com>.
I will reach out right now
From: Simon McGloin [mailto:simonmcgloin@gmail.com]
Sent: Friday, October 18, 2013 12:24 PM
To: user@giraph.apache.org
Subject: Re: Master always fails on dataset
Thanks Claudio. Yes the machines are homogenous. Unfortunately I don't have ganglia installed. You were right it is a memory issue. I've reduced the number of partitions down to 1 with -Dgiraph.maxPartitionsInMemory=1 and now my jobs are failing due to running out of diskspace on HDFS. Each HDFS mount has 100gb of space. I will increase the size of HDFS and order more memory next week. Is there anyway to calculate the memory requirements of a giraph job? I presume it depends on the algorithm being run.
On Thu, Oct 17, 2013 at 6:42 PM, Claudio Martella <cl...@gmail.com>> wrote:
Try decreasing the number of partitions you keep in memory. You're running out of memory. Also, are your nodes homogenous? It could be one particular machine swapping or something. If you have ganglia, try investigating the usage of memory.
On Thu, Oct 17, 2013 at 7:39 PM, Simon McGloin <si...@gmail.com>> wrote:
Hey Guys.
I have a problem running my giraph job on a dataset with 20,000,000 edges and 2,000,000 vertices. All the vertices are Text based. The giraph job works perfectly on smaller datasets but always fails on larger ones. The setup I have is a 3 node cluster, each with 24 cores and 24 GB of ram. The cluster has a total of 60 mappers each with mapred.child.java.opts set to -Xmx1000m.
If I don't use the Out-of-Core option then the job fails due to running out of java heap space. When I use -Dgiraph.useOutOfCoreGraph=true then the master eventually fails due to a worker disconnecting from zookeeper. The worker just throws a warning and doesn't actually fail. I've been using the -Dgiraph.checkpointFrequency=1 option but this doesn't seem to restart the mapper. I'm new to zookeeper too so if this is a zookeeper problem then let me know and I can investigate it as such.
Below is the options I'm using and the errors I'm currently getting
Any help or tips are appreciated,
Simon
Options:
-Dgiraph.zkList=10.10.5.103:2181<http://10.10.5.103:2181>,10.10.5.104:2181<http://10.10.5.104:2181>,10.10.5.105:2181<http://10.10.5.105:2181>
-Dgiraph.checkpointFrequency=1
-Dgiraph.useOutOfCoreGraph=true
-Dgiraph.zkSessionMsecTimeout=600000
-Dgiraph.numComputeThreads=2
Master Log:
2013-10-17 18:19:34,638 INFO org.apache.giraph.master.BspServiceMaster: barrierOnWorkerList: 0 out of 50 workers finished on superstep 1 on path /_hadoopBsp/job_201310161506_0064/_applicationAttemptsDir/0/_superstepDir/1/_workerWroteCheckpointDir
2013-10-17 18:20:52,105 ERROR org.apache.giraph.master.BspServiceMaster: superstepChosenWorkerAlive: Missing chosen worker Worker(hostname=node1.mycompany.com<http://node1.mycompany.com>, MRtaskID=30, port=30030) on superstep 1
2013-10-17 18:20:52,106 INFO org.apache.giraph.master.MasterThread: masterThread: Coordination of superstep 1 took 78.851 seconds ended with state WORKER_FAILURE and is now on superstep 1
2013-10-17 18:20:52,112 ERROR org.apache.giraph.master.MasterThread: masterThread: Master algorithm failed with RuntimeException
java.lang.RuntimeException: restartFromCheckpoint: KeeperException
at org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1185)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:135)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /_hadoopBsp/job_201310161506_0064/_vertexInputSplitDir
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:728)
at org.apache.giraph.zk.ZooKeeperExt.deleteExt(ZooKeeperExt.java:307)
at org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1177)
... 1 more
2013-10-17 18:20:52,115 FATAL org.apache.giraph.graph.GraphMapper: uncaughtException: OverrideExceptionHandler on thread org.apache.giraph.master.MasterThread, msg = java.lang.RuntimeException: restartFromCheckpoint: KeeperException, exiting...
java.lang.IllegalStateException: java.lang.RuntimeException: restartFromCheckpoint: KeeperException
at org.apache.giraph.master.MasterThread.run(MasterThread.java:181)
Caused by: java.lang.RuntimeException: restartFromCheckpoint: KeeperException
at org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1185)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:135)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /_hadoopBsp/job_201310161506_0064/_vertexInputSplitDir
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:728)
at org.apache.giraph.zk.ZooKeeperExt.deleteExt(ZooKeeperExt.java:307)
at org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1177)
... 1 more
Worker 30 log:
2013-10-17 18:19:07,309 INFO org.apache.giraph.partition.DiskBackedPartitionStore: offloadPartition: writing partition edges 1927 to /data/var/hdfs/data/mapred/taskTracker/simon/jobcache/job_201310161506_0064/attempt_201310161506_0064_m_000030_0/work/_bsp/_partitions/job_201310161506_0064/partition-1927_edges
2013-10-17 18:19:45,736 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Future result not ready yet java.util.concurrent.FutureTask@c07bacb
2013-10-17 18:19:45,737 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@4f786b98
2013-10-17 18:19:45,789 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 40183ms for sessionid 0x341c716ad860073, closing socket connection and attempting reconnect
2013-10-17 18:19:46,113 WARN org.apache.giraph.bsp.BspService: process: Disconnected from ZooKeeper (will automatically try to recover) WatchedEvent state:Disconnected type:None path:null
2013-10-17 18:19:46,113 WARN org.apache.giraph.worker.InputSplitsHandler: process: Problem with zookeeper, got event with path null, state Disconnected, event type None
2013-10-17 18:19:46,746 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.10.5.105:2181<http://10.10.5.105:2181>
2013-10-17 18:19:46,747 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to node3.mycompany.com/10.10.5.105:2181<http://node3.mycompany.com/10.10.5.105:2181>, initiating session
2013-10-17 18:19:46,750 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x341c716ad860073 has expired, closing socket connection
2013-10-17 18:19:46,750 WARN org.apache.giraph.bsp.BspService: process: Got unknown null path event WatchedEvent state:Expired type:None path:null
2013-10-17 18:19:46,750 WARN org.apache.giraph.worker.InputSplitsHandler: process: Problem with zookeeper, got event with path null, state Expired, event type None
2013-10-17 18:19:46,750 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2013-10-17 18:20:33,546 INFO org.apache.giraph.comm.netty.handler.RequestDecoder: decode: Server window metrics MBytes/sec sent = 0, MBytes/sec received = 0.0059, MBytesSent = 0.0008, MBytesReceived = 0.7636, ave sent req MBytes = 0, ave received req MBytes = 0.0111, secs waited = 128.396
2013-10-17 18:20:45,737 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Future result not ready yet java.util.concurrent.FutureTask@c07bacb
2013-10-17 18:20:45,737 INFO org.apache.giraph.utils.ProgressableUtils: waitFor: Waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@4f786b98
--
Claudio Martella
claudio.martella@gmail.com<ma...@gmail.com>
Re: Master always fails on dataset
Posted by Simon McGloin <si...@gmail.com>.
Thanks Claudio. Yes the machines are homogenous. Unfortunately I don't have
ganglia installed. You were right it is a memory issue. I've reduced the
number of partitions down to 1 with -Dgiraph.maxPartitionsInMemory=1 and
now my jobs are failing due to running out of diskspace on HDFS. Each HDFS
mount has 100gb of space. I will increase the size of HDFS and order more
memory next week. Is there anyway to calculate the memory requirements of a
giraph job? I presume it depends on the algorithm being run.
On Thu, Oct 17, 2013 at 6:42 PM, Claudio Martella <
claudio.martella@gmail.com> wrote:
> Try decreasing the number of partitions you keep in memory. You're running
> out of memory. Also, are your nodes homogenous? It could be one particular
> machine swapping or something. If you have ganglia, try investigating the
> usage of memory.
>
>
> On Thu, Oct 17, 2013 at 7:39 PM, Simon McGloin <si...@gmail.com>wrote:
>
>> Hey Guys.
>>
>> I have a problem running my giraph job on a dataset with 20,000,000 edges
>> and 2,000,000 vertices. All the vertices are Text based. The giraph job
>> works perfectly on smaller datasets but always fails on larger ones. The
>> setup I have is a 3 node cluster, each with 24 cores and 24 GB of ram. The
>> cluster has a total of 60 mappers each with mapred.child.java.opts set to
>> -Xmx1000m.
>> If I don't use the Out-of-Core option then the job fails due to running
>> out of java heap space. When I use -Dgiraph.useOutOfCoreGraph=true then the
>> master eventually fails due to a worker disconnecting from zookeeper. The
>> worker just throws a warning and doesn't actually fail. I've been using the
>> -Dgiraph.checkpointFrequency=1 option but this doesn't seem to restart the
>> mapper. I'm new to zookeeper too so if this is a zookeeper problem then let
>> me know and I can investigate it as such.
>>
>> Below is the options I'm using and the errors I'm currently getting
>> Any help or tips are appreciated,
>> Simon
>>
>> Options:
>> -Dgiraph.zkList=10.10.5.103:2181,10.10.5.104:2181,10.10.5.105:2181
>> -Dgiraph.checkpointFrequency=1
>> -Dgiraph.useOutOfCoreGraph=true
>> -Dgiraph.zkSessionMsecTimeout=600000
>> -Dgiraph.numComputeThreads=2
>>
>> Master Log:
>> 2013-10-17 18:19:34,638 INFO org.apache.giraph.master.BspServiceMaster:
>> barrierOnWorkerList: 0 out of 50 workers finished on superstep 1 on path
>> /_hadoopBsp/job_201310161506_0064/_applicationAttemptsDir/0/_superstepDir/1/_workerWroteCheckpointDir
>> 2013-10-17 18:20:52,105 ERROR org.apache.giraph.master.BspServiceMaster:
>> superstepChosenWorkerAlive: Missing chosen worker Worker(hostname=
>> node1.mycompany.com, MRtaskID=30, port=30030) on superstep 1
>> 2013-10-17 18:20:52,106 INFO org.apache.giraph.master.MasterThread:
>> masterThread: Coordination of superstep 1 took 78.851 seconds ended with
>> state WORKER_FAILURE and is now on superstep 1
>> 2013-10-17 18:20:52,112 ERROR org.apache.giraph.master.MasterThread:
>> masterThread: Master algorithm failed with RuntimeException
>> java.lang.RuntimeException: restartFromCheckpoint: KeeperException
>> at
>> org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1185)
>> at org.apache.giraph.master.MasterThread.run(MasterThread.java:135)
>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
>> KeeperErrorCode = NoNode for
>> /_hadoopBsp/job_201310161506_0064/_vertexInputSplitDir
>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:728)
>> at org.apache.giraph.zk.ZooKeeperExt.deleteExt(ZooKeeperExt.java:307)
>> at
>> org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1177)
>> ... 1 more
>> 2013-10-17 18:20:52,115 FATAL org.apache.giraph.graph.GraphMapper:
>> uncaughtException: OverrideExceptionHandler on thread
>> org.apache.giraph.master.MasterThread, msg = java.lang.RuntimeException:
>> restartFromCheckpoint: KeeperException, exiting...
>> java.lang.IllegalStateException: java.lang.RuntimeException:
>> restartFromCheckpoint: KeeperException
>> at org.apache.giraph.master.MasterThread.run(MasterThread.java:181)
>> Caused by: java.lang.RuntimeException: restartFromCheckpoint:
>> KeeperException
>> at
>> org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1185)
>> at org.apache.giraph.master.MasterThread.run(MasterThread.java:135)
>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
>> KeeperErrorCode = NoNode for
>> /_hadoopBsp/job_201310161506_0064/_vertexInputSplitDir
>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:728)
>> at org.apache.giraph.zk.ZooKeeperExt.deleteExt(ZooKeeperExt.java:307)
>> at
>> org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1177)
>> ... 1 more
>>
>>
>> Worker 30 log:
>> 2013-10-17 18:19:07,309 INFO
>> org.apache.giraph.partition.DiskBackedPartitionStore: offloadPartition:
>> writing partition edges 1927 to
>> /data/var/hdfs/data/mapred/taskTracker/simon/jobcache/job_201310161506_0064/attempt_201310161506_0064_m_000030_0/work/_bsp/_partitions/job_201310161506_0064/partition-1927_edges
>> 2013-10-17 18:19:45,736 INFO org.apache.giraph.utils.ProgressableUtils:
>> waitFor: Future result not ready yet java.util.concurrent.FutureTask@c07bacb
>> 2013-10-17 18:19:45,737 INFO org.apache.giraph.utils.ProgressableUtils:
>> waitFor: Waiting for
>> org.apache.giraph.utils.ProgressableUtils$FutureWaitable@4f786b98
>> 2013-10-17 18:19:45,789 INFO org.apache.zookeeper.ClientCnxn: Client
>> session timed out, have not heard from server in 40183ms for sessionid
>> 0x341c716ad860073, closing socket connection and attempting reconnect
>> 2013-10-17 18:19:46,113 WARN org.apache.giraph.bsp.BspService: process:
>> Disconnected from ZooKeeper (will automatically try to recover)
>> WatchedEvent state:Disconnected type:None path:null
>> 2013-10-17 18:19:46,113 WARN org.apache.giraph.worker.InputSplitsHandler:
>> process: Problem with zookeeper, got event with path null, state
>> Disconnected, event type None
>> 2013-10-17 18:19:46,746 INFO org.apache.zookeeper.ClientCnxn: Opening
>> socket connection to server /10.10.5.105:2181
>> 2013-10-17 18:19:46,747 INFO org.apache.zookeeper.ClientCnxn: Socket
>> connection established to node3.mycompany.com/10.10.5.105:2181,
>> initiating session
>> 2013-10-17 18:19:46,750 INFO org.apache.zookeeper.ClientCnxn: Unable to
>> reconnect to ZooKeeper service, session 0x341c716ad860073 has expired,
>> closing socket connection
>> 2013-10-17 18:19:46,750 WARN org.apache.giraph.bsp.BspService: process:
>> Got unknown null path event WatchedEvent state:Expired type:None path:null
>> 2013-10-17 18:19:46,750 WARN org.apache.giraph.worker.InputSplitsHandler:
>> process: Problem with zookeeper, got event with path null, state Expired,
>> event type None
>> 2013-10-17 18:19:46,750 INFO org.apache.zookeeper.ClientCnxn: EventThread
>> shut down
>> 2013-10-17 18:20:33,546 INFO
>> org.apache.giraph.comm.netty.handler.RequestDecoder: decode: Server window
>> metrics MBytes/sec sent = 0, MBytes/sec received = 0.0059, MBytesSent =
>> 0.0008, MBytesReceived = 0.7636, ave sent req MBytes = 0, ave received req
>> MBytes = 0.0111, secs waited = 128.396
>> 2013-10-17 18:20:45,737 INFO org.apache.giraph.utils.ProgressableUtils:
>> waitFor: Future result not ready yet java.util.concurrent.FutureTask@c07bacb
>> 2013-10-17 18:20:45,737 INFO org.apache.giraph.utils.ProgressableUtils:
>> waitFor: Waiting for
>> org.apache.giraph.utils.ProgressableUtils$FutureWaitable@4f786b98
>>
>>
>>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com
>
Re: Master always fails on dataset
Posted by Claudio Martella <cl...@gmail.com>.
Try decreasing the number of partitions you keep in memory. You're running
out of memory. Also, are your nodes homogenous? It could be one particular
machine swapping or something. If you have ganglia, try investigating the
usage of memory.
On Thu, Oct 17, 2013 at 7:39 PM, Simon McGloin <si...@gmail.com>wrote:
> Hey Guys.
>
> I have a problem running my giraph job on a dataset with 20,000,000 edges
> and 2,000,000 vertices. All the vertices are Text based. The giraph job
> works perfectly on smaller datasets but always fails on larger ones. The
> setup I have is a 3 node cluster, each with 24 cores and 24 GB of ram. The
> cluster has a total of 60 mappers each with mapred.child.java.opts set to
> -Xmx1000m.
> If I don't use the Out-of-Core option then the job fails due to running
> out of java heap space. When I use -Dgiraph.useOutOfCoreGraph=true then the
> master eventually fails due to a worker disconnecting from zookeeper. The
> worker just throws a warning and doesn't actually fail. I've been using the
> -Dgiraph.checkpointFrequency=1 option but this doesn't seem to restart the
> mapper. I'm new to zookeeper too so if this is a zookeeper problem then let
> me know and I can investigate it as such.
>
> Below is the options I'm using and the errors I'm currently getting
> Any help or tips are appreciated,
> Simon
>
> Options:
> -Dgiraph.zkList=10.10.5.103:2181,10.10.5.104:2181,10.10.5.105:2181
> -Dgiraph.checkpointFrequency=1
> -Dgiraph.useOutOfCoreGraph=true
> -Dgiraph.zkSessionMsecTimeout=600000
> -Dgiraph.numComputeThreads=2
>
> Master Log:
> 2013-10-17 18:19:34,638 INFO org.apache.giraph.master.BspServiceMaster:
> barrierOnWorkerList: 0 out of 50 workers finished on superstep 1 on path
> /_hadoopBsp/job_201310161506_0064/_applicationAttemptsDir/0/_superstepDir/1/_workerWroteCheckpointDir
> 2013-10-17 18:20:52,105 ERROR org.apache.giraph.master.BspServiceMaster:
> superstepChosenWorkerAlive: Missing chosen worker Worker(hostname=
> node1.mycompany.com, MRtaskID=30, port=30030) on superstep 1
> 2013-10-17 18:20:52,106 INFO org.apache.giraph.master.MasterThread:
> masterThread: Coordination of superstep 1 took 78.851 seconds ended with
> state WORKER_FAILURE and is now on superstep 1
> 2013-10-17 18:20:52,112 ERROR org.apache.giraph.master.MasterThread:
> masterThread: Master algorithm failed with RuntimeException
> java.lang.RuntimeException: restartFromCheckpoint: KeeperException
> at
> org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1185)
> at org.apache.giraph.master.MasterThread.run(MasterThread.java:135)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for
> /_hadoopBsp/job_201310161506_0064/_vertexInputSplitDir
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:728)
> at org.apache.giraph.zk.ZooKeeperExt.deleteExt(ZooKeeperExt.java:307)
> at
> org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1177)
> ... 1 more
> 2013-10-17 18:20:52,115 FATAL org.apache.giraph.graph.GraphMapper:
> uncaughtException: OverrideExceptionHandler on thread
> org.apache.giraph.master.MasterThread, msg = java.lang.RuntimeException:
> restartFromCheckpoint: KeeperException, exiting...
> java.lang.IllegalStateException: java.lang.RuntimeException:
> restartFromCheckpoint: KeeperException
> at org.apache.giraph.master.MasterThread.run(MasterThread.java:181)
> Caused by: java.lang.RuntimeException: restartFromCheckpoint:
> KeeperException
> at
> org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1185)
> at org.apache.giraph.master.MasterThread.run(MasterThread.java:135)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for
> /_hadoopBsp/job_201310161506_0064/_vertexInputSplitDir
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:728)
> at org.apache.giraph.zk.ZooKeeperExt.deleteExt(ZooKeeperExt.java:307)
> at
> org.apache.giraph.master.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1177)
> ... 1 more
>
>
> Worker 30 log:
> 2013-10-17 18:19:07,309 INFO
> org.apache.giraph.partition.DiskBackedPartitionStore: offloadPartition:
> writing partition edges 1927 to
> /data/var/hdfs/data/mapred/taskTracker/simon/jobcache/job_201310161506_0064/attempt_201310161506_0064_m_000030_0/work/_bsp/_partitions/job_201310161506_0064/partition-1927_edges
> 2013-10-17 18:19:45,736 INFO org.apache.giraph.utils.ProgressableUtils:
> waitFor: Future result not ready yet java.util.concurrent.FutureTask@c07bacb
> 2013-10-17 18:19:45,737 INFO org.apache.giraph.utils.ProgressableUtils:
> waitFor: Waiting for
> org.apache.giraph.utils.ProgressableUtils$FutureWaitable@4f786b98
> 2013-10-17 18:19:45,789 INFO org.apache.zookeeper.ClientCnxn: Client
> session timed out, have not heard from server in 40183ms for sessionid
> 0x341c716ad860073, closing socket connection and attempting reconnect
> 2013-10-17 18:19:46,113 WARN org.apache.giraph.bsp.BspService: process:
> Disconnected from ZooKeeper (will automatically try to recover)
> WatchedEvent state:Disconnected type:None path:null
> 2013-10-17 18:19:46,113 WARN org.apache.giraph.worker.InputSplitsHandler:
> process: Problem with zookeeper, got event with path null, state
> Disconnected, event type None
> 2013-10-17 18:19:46,746 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server /10.10.5.105:2181
> 2013-10-17 18:19:46,747 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to node3.mycompany.com/10.10.5.105:2181,
> initiating session
> 2013-10-17 18:19:46,750 INFO org.apache.zookeeper.ClientCnxn: Unable to
> reconnect to ZooKeeper service, session 0x341c716ad860073 has expired,
> closing socket connection
> 2013-10-17 18:19:46,750 WARN org.apache.giraph.bsp.BspService: process:
> Got unknown null path event WatchedEvent state:Expired type:None path:null
> 2013-10-17 18:19:46,750 WARN org.apache.giraph.worker.InputSplitsHandler:
> process: Problem with zookeeper, got event with path null, state Expired,
> event type None
> 2013-10-17 18:19:46,750 INFO org.apache.zookeeper.ClientCnxn: EventThread
> shut down
> 2013-10-17 18:20:33,546 INFO
> org.apache.giraph.comm.netty.handler.RequestDecoder: decode: Server window
> metrics MBytes/sec sent = 0, MBytes/sec received = 0.0059, MBytesSent =
> 0.0008, MBytesReceived = 0.7636, ave sent req MBytes = 0, ave received req
> MBytes = 0.0111, secs waited = 128.396
> 2013-10-17 18:20:45,737 INFO org.apache.giraph.utils.ProgressableUtils:
> waitFor: Future result not ready yet java.util.concurrent.FutureTask@c07bacb
> 2013-10-17 18:20:45,737 INFO org.apache.giraph.utils.ProgressableUtils:
> waitFor: Waiting for
> org.apache.giraph.utils.ProgressableUtils$FutureWaitable@4f786b98
>
>
>
--
Claudio Martella
claudio.martella@gmail.com