You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by dlogothetis <gi...@git.apache.org> on 2018/09/18 16:34:45 UTC
[GitHub] giraph pull request #84: Fix BspServiceMaster bug
GitHub user dlogothetis opened a pull request:
https://github.com/apache/giraph/pull/84
Fix BspServiceMaster bug
Occasionally, BspServiceMaster fails to detect that a worker finished a superstep (e.g. input superstep, or a compute supestep). As a result, the job may get stuck as the master keeps waiting for all workers to finish.
While this is hard to reproduce, this fixes a potential cause for this.
Tests
- Unit tests
- Internal snapshot tests
- Ran large job
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dlogothetis/giraph fix_bsp
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/giraph/pull/84.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #84
----
commit 507956f1a053510be2e4d41cdf0d3444bddd4bbb
Author: Dionysios Logothetis <dl...@...>
Date: 2018-09-18T16:30:20Z
Fix BspServiceMaster bug
----
---
[GitHub] giraph pull request #84: Fix BspServiceMaster bug
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/giraph/pull/84
---
[GitHub] giraph pull request #84: Fix BspServiceMaster bug
Posted by dlogothetis <gi...@git.apache.org>.
Github user dlogothetis commented on a diff in the pull request:
https://github.com/apache/giraph/pull/84#discussion_r218556285
--- Diff: giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java ---
@@ -1379,9 +1379,15 @@ private boolean barrierOnWorkerList(String finishedWorkerPath,
// Wait for a signal or timeout
boolean eventTriggered = event.waitMsecs(eventLoopTimeout);
+
+ // If the event was triggered, we reset it. In the next loop run, we will
+ // read ZK to get the new hosts.
+ if (eventTriggered) {
+ event.reset();
+ }
+
long elapsedTimeSinceRegularRunMsec = System.currentTimeMillis() -
lastRegularRunTimeMsec;
- event.reset();
--- End diff --
Are you talking about re-ordering the `long elapsedTimeSinceRegularRunMsec = ` command and the `event.reset()` command? That shouldn't matter, I just moved the `if(eventTriggered)` because is seemed a bit clearer to put it close to the code that outputs `eventTriggered`. The fix is the actual `if(eventTriggered)`.
---
[GitHub] giraph pull request #84: Fix BspServiceMaster bug
Posted by dlogothetis <gi...@git.apache.org>.
Github user dlogothetis commented on a diff in the pull request:
https://github.com/apache/giraph/pull/84#discussion_r218507979
--- Diff: giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java ---
@@ -1379,9 +1379,15 @@ private boolean barrierOnWorkerList(String finishedWorkerPath,
// Wait for a signal or timeout
boolean eventTriggered = event.waitMsecs(eventLoopTimeout);
+
+ // If the event was triggered, we reset it. In the next loop run, we will
+ // read ZK to get the new hosts.
+ if (eventTriggered) {
+ event.reset();
+ }
+
long elapsedTimeSinceRegularRunMsec = System.currentTimeMillis() -
lastRegularRunTimeMsec;
- event.reset();
--- End diff --
It's possible that after `event.waitMsecs` exits (due to timeout) and before `event.reset()` get called, the event get signaled. In this case, in the next loop `event.waitMsec` will timeout again and `logInfoOnlyRun` will continue to be false.
---
[GitHub] giraph pull request #84: Fix BspServiceMaster bug
Posted by majakabiljo <gi...@git.apache.org>.
Github user majakabiljo commented on a diff in the pull request:
https://github.com/apache/giraph/pull/84#discussion_r218551052
--- Diff: giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java ---
@@ -1379,9 +1379,15 @@ private boolean barrierOnWorkerList(String finishedWorkerPath,
// Wait for a signal or timeout
boolean eventTriggered = event.waitMsecs(eventLoopTimeout);
+
+ // If the event was triggered, we reset it. In the next loop run, we will
+ // read ZK to get the new hosts.
+ if (eventTriggered) {
+ event.reset();
+ }
+
long elapsedTimeSinceRegularRunMsec = System.currentTimeMillis() -
lastRegularRunTimeMsec;
- event.reset();
--- End diff --
Making sure I understand what's happening, does reodering of commands here makes no difference and the fix is about only clearing event if it triggered?
---