You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by "Jose Luis Larroque (JIRA)" <ji...@apache.org> on 2016/07/31 01:51:20 UTC
[jira] [Updated] (GIRAPH-1101) Giraph hangs indefinitely when two
or more workers process the same vertice on the same superstep
[ https://issues.apache.org/jira/browse/GIRAPH-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jose Luis Larroque updated GIRAPH-1101:
---------------------------------------
Description:
If two workers (or more) are proccesing the same vertice on same superstep (for example, doing mulple BFS at the same time, could lead to it, depending of the data of course), the entire superstep hangs, every workers say something like this:
16/07/29 22:49:19 INFO utils.ProgressableUtils: waitFor: Future result not ready yet java.util.concurrent.FutureTask@23a1ef14
16/07/29 22:49:19 INFO utils.ProgressableUtils: waitFor: Waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@5c571c52
16/07/29 22:50:19 INFO utils.ProgressableUtils: waitFor: Future result not ready yet java.util.concurrent.FutureTask@23a1ef14
And the master says:
{quote}
16/07/29 21:43:19 INFO yarn.GiraphYarnTask: [STATUS: task-0] MASTER_ZOOKEEPER_ONLY - 0 finished out of 4 on superstep 4
16/07/29 21:43:19 DEBUG master.BspServiceMaster: barrierOnWorkerList: Got finished worker list = [], size = 0, worker list = [Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal, MRtaskID=1, port=30001), Worker(hostname=ip-172-31-23-12.sa-east-1.compute.internal, MRtaskID=2, port=30002), Worker(hostname=ip-172-31-23-11.sa-east-1.compute.internal, MRtaskID=3, port=30003), Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal, MRtaskID=5, port=30005)], size = 4 from /_hadoopBsp/giraph_yarn_application_1469827475142_0001/_applicationAttemptsDir/0/_superstepDir/4/_workerFinishedDir
16/07/29 21:43:19 INFO yarn.GiraphYarnTask: [STATUS: task-0] MASTER_ZOOKEEPER_ONLY - 0 finished out of 4 on superstep 4
16/07/29 21:43:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:43:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:43:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:43:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:43:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:43:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:43:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:43:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:43:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
{/quote}
16/07/29 21:44:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:48:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:48:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:48:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:48:19 DEBUG master.BspServiceMaster: barrierOnWorkerList: Got finished worker list = [], size = 0, worker list = [Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal, MRtaskID=1, port=30001), Worker(hostname=ip-172-31-23-12.sa-east-1.compute.internal, MRtaskID=2, port=30002), Worker(hostname=ip-172-31-23-11.sa-east-1.compute.internal, MRtaskID=3, port=30003), Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal, MRtaskID=5, port=30005)], size = 4 from /_hadoopBsp/giraph_yarn_application_1469827475142_0001/_applicationAttemptsDir/0/_superstepDir/4/_workerFinishedDir
16/07/29 21:48:19 INFO master.BspServiceMaster: barrierOnWorkerList: 0 out of 4 workers finished on superstep 4 on path /_hadoopBsp/giraph_yarn_application_1469827475142_0001/_applicationAttemptsDir/0/_superstepDir/4/_workerFinishedDir
16/07/29 21:48:19 INFO master.BspServiceMaster: barrierOnWorkerList: Waiting on [ip-172-31-23-12.sa-east-1.compute.internal_2, ip-172-31-23-9.sa-east-1.compute.internal_5, ip-172-31-23-11.sa-east-1.compute.internal_3, ip-172-31-23-9.sa-east-1.compute.internal_1]
16/07/29 21:48:19 INFO yarn.GiraphYarnTask: [STATUS: task-0] MASTER_ZOOKEEPER_ONLY - 0 finished out of 4 on superstep 4
16/07/29 21:48:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:48:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:48:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:48:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:48:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 22:50:19 INFO utils.ProgressableUtils: waitFor: Waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@5c571c52
was:
If two workers (or more) are proccesing the same vertice on same superstep (for example, doing mulple BFS at the same time, could lead to it, depending of the data of course), the entire superstep hangs, every workers say something like this:
16/07/29 22:49:19 INFO utils.ProgressableUtils: waitFor: Future result not ready yet java.util.concurrent.FutureTask@23a1ef14
16/07/29 22:49:19 INFO utils.ProgressableUtils: waitFor: Waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@5c571c52
16/07/29 22:50:19 INFO utils.ProgressableUtils: waitFor: Future result not ready yet java.util.concurrent.FutureTask@23a1ef14
And the master says:
16/07/29 21:43:19 INFO yarn.GiraphYarnTask: [STATUS: task-0] MASTER_ZOOKEEPER_ONLY - 0 finished out of 4 on superstep 4
16/07/29 21:43:19 DEBUG master.BspServiceMaster: barrierOnWorkerList: Got finished worker list = [], size = 0, worker list = [Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal, MRtaskID=1, port=30001), Worker(hostname=ip-172-31-23-12.sa-east-1.compute.internal, MRtaskID=2, port=30002), Worker(hostname=ip-172-31-23-11.sa-east-1.compute.internal, MRtaskID=3, port=30003), Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal, MRtaskID=5, port=30005)], size = 4 from /_hadoopBsp/giraph_yarn_application_1469827475142_0001/_applicationAttemptsDir/0/_superstepDir/4/_workerFinishedDir
16/07/29 21:43:19 INFO yarn.GiraphYarnTask: [STATUS: task-0] MASTER_ZOOKEEPER_ONLY - 0 finished out of 4 on superstep 4
16/07/29 21:43:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:43:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:43:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:43:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:43:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:43:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:43:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:43:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:43:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:44:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:44:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:45:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:45:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:46:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:46:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:47:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:47:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:48:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:48:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:48:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:48:19 DEBUG master.BspServiceMaster: barrierOnWorkerList: Got finished worker list = [], size = 0, worker list = [Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal, MRtaskID=1, port=30001), Worker(hostname=ip-172-31-23-12.sa-east-1.compute.internal, MRtaskID=2, port=30002), Worker(hostname=ip-172-31-23-11.sa-east-1.compute.internal, MRtaskID=3, port=30003), Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal, MRtaskID=5, port=30005)], size = 4 from /_hadoopBsp/giraph_yarn_application_1469827475142_0001/_applicationAttemptsDir/0/_superstepDir/4/_workerFinishedDir
16/07/29 21:48:19 INFO master.BspServiceMaster: barrierOnWorkerList: 0 out of 4 workers finished on superstep 4 on path /_hadoopBsp/giraph_yarn_application_1469827475142_0001/_applicationAttemptsDir/0/_superstepDir/4/_workerFinishedDir
16/07/29 21:48:19 INFO master.BspServiceMaster: barrierOnWorkerList: Waiting on [ip-172-31-23-12.sa-east-1.compute.internal_2, ip-172-31-23-9.sa-east-1.compute.internal_5, ip-172-31-23-11.sa-east-1.compute.internal_3, ip-172-31-23-9.sa-east-1.compute.internal_1]
16/07/29 21:48:19 INFO yarn.GiraphYarnTask: [STATUS: task-0] MASTER_ZOOKEEPER_ONLY - 0 finished out of 4 on superstep 4
16/07/29 21:48:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:48:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:48:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 21:48:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
16/07/29 21:48:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
16/07/29 22:50:19 INFO utils.ProgressableUtils: waitFor: Waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@5c571c52
> Giraph hangs indefinitely when two or more workers process the same vertice on the same superstep
> -------------------------------------------------------------------------------------------------
>
> Key: GIRAPH-1101
> URL: https://issues.apache.org/jira/browse/GIRAPH-1101
> Project: Giraph
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Jose Luis Larroque
> Priority: Minor
>
> If two workers (or more) are proccesing the same vertice on same superstep (for example, doing mulple BFS at the same time, could lead to it, depending of the data of course), the entire superstep hangs, every workers say something like this:
> 16/07/29 22:49:19 INFO utils.ProgressableUtils: waitFor: Future result not ready yet java.util.concurrent.FutureTask@23a1ef14
> 16/07/29 22:49:19 INFO utils.ProgressableUtils: waitFor: Waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@5c571c52
> 16/07/29 22:50:19 INFO utils.ProgressableUtils: waitFor: Future result not ready yet java.util.concurrent.FutureTask@23a1ef14
> And the master says:
> {quote}
> 16/07/29 21:43:19 INFO yarn.GiraphYarnTask: [STATUS: task-0] MASTER_ZOOKEEPER_ONLY - 0 finished out of 4 on superstep 4
> 16/07/29 21:43:19 DEBUG master.BspServiceMaster: barrierOnWorkerList: Got finished worker list = [], size = 0, worker list = [Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal, MRtaskID=1, port=30001), Worker(hostname=ip-172-31-23-12.sa-east-1.compute.internal, MRtaskID=2, port=30002), Worker(hostname=ip-172-31-23-11.sa-east-1.compute.internal, MRtaskID=3, port=30003), Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal, MRtaskID=5, port=30005)], size = 4 from /_hadoopBsp/giraph_yarn_application_1469827475142_0001/_applicationAttemptsDir/0/_superstepDir/4/_workerFinishedDir
> 16/07/29 21:43:19 INFO yarn.GiraphYarnTask: [STATUS: task-0] MASTER_ZOOKEEPER_ONLY - 0 finished out of 4 on superstep 4
> 16/07/29 21:43:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:43:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:43:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:43:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:43:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:43:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:43:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:43:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:43:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:44:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:44:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:44:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:44:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:44:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:44:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:44:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:44:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:44:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:44:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> {/quote}
> 16/07/29 21:44:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:44:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:45:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:45:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:45:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:45:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:45:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:45:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:45:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:45:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:45:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:45:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:45:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:45:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:46:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:46:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:46:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:46:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:46:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:46:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:46:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:46:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:46:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:46:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:46:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:46:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:47:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:47:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:47:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:47:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:47:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:47:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:47:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:47:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:47:49 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:47:49 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:47:59 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:47:59 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:48:09 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:48:09 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:48:19 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:48:19 DEBUG master.BspServiceMaster: barrierOnWorkerList: Got finished worker list = [], size = 0, worker list = [Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal, MRtaskID=1, port=30001), Worker(hostname=ip-172-31-23-12.sa-east-1.compute.internal, MRtaskID=2, port=30002), Worker(hostname=ip-172-31-23-11.sa-east-1.compute.internal, MRtaskID=3, port=30003), Worker(hostname=ip-172-31-23-9.sa-east-1.compute.internal, MRtaskID=5, port=30005)], size = 4 from /_hadoopBsp/giraph_yarn_application_1469827475142_0001/_applicationAttemptsDir/0/_superstepDir/4/_workerFinishedDir
> 16/07/29 21:48:19 INFO master.BspServiceMaster: barrierOnWorkerList: 0 out of 4 workers finished on superstep 4 on path /_hadoopBsp/giraph_yarn_application_1469827475142_0001/_applicationAttemptsDir/0/_superstepDir/4/_workerFinishedDir
> 16/07/29 21:48:19 INFO master.BspServiceMaster: barrierOnWorkerList: Waiting on [ip-172-31-23-12.sa-east-1.compute.internal_2, ip-172-31-23-9.sa-east-1.compute.internal_5, ip-172-31-23-11.sa-east-1.compute.internal_3, ip-172-31-23-9.sa-east-1.compute.internal_1]
> 16/07/29 21:48:19 INFO yarn.GiraphYarnTask: [STATUS: task-0] MASTER_ZOOKEEPER_ONLY - 0 finished out of 4 on superstep 4
> 16/07/29 21:48:19 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:48:29 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:48:29 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 21:48:39 DEBUG zk.PredicateLock: waitMsecs: Got timed signaled of false
> 16/07/29 21:48:39 DEBUG zk.PredicateLock: waitMsecs: Wait for 10000
> 16/07/29 22:50:19 INFO utils.ProgressableUtils: waitFor: Waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@5c571c52
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)