You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by dlogothetis <gi...@git.apache.org> on 2018/09/18 16:34:45 UTC

[GitHub] giraph pull request #84: Fix BspServiceMaster bug

GitHub user dlogothetis opened a pull request:

    https://github.com/apache/giraph/pull/84

    Fix BspServiceMaster bug

    Occasionally, BspServiceMaster fails to detect that a worker finished a superstep (e.g. input superstep, or a compute supestep). As a result, the job may get stuck as the master keeps waiting for all workers to finish.
    
    While this is hard to reproduce, this fixes a potential cause for this.
    
    Tests
    - Unit tests
    - Internal snapshot tests
    - Ran large job

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dlogothetis/giraph fix_bsp

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/giraph/pull/84.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #84
    
----
commit 507956f1a053510be2e4d41cdf0d3444bddd4bbb
Author: Dionysios Logothetis <dl...@...>
Date:   2018-09-18T16:30:20Z

    Fix BspServiceMaster bug

----


---

[GitHub] giraph pull request #84: Fix BspServiceMaster bug

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/giraph/pull/84


---

[GitHub] giraph pull request #84: Fix BspServiceMaster bug

Posted by dlogothetis <gi...@git.apache.org>.
Github user dlogothetis commented on a diff in the pull request:

    https://github.com/apache/giraph/pull/84#discussion_r218556285
  
    --- Diff: giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java ---
    @@ -1379,9 +1379,15 @@ private boolean barrierOnWorkerList(String finishedWorkerPath,
     
           // Wait for a signal or timeout
           boolean eventTriggered = event.waitMsecs(eventLoopTimeout);
    +
    +      // If the event was triggered, we reset it. In the next loop run, we will
    +      // read ZK to get the new hosts.
    +      if (eventTriggered) {
    +        event.reset();
    +      }
    +
           long elapsedTimeSinceRegularRunMsec = System.currentTimeMillis() -
               lastRegularRunTimeMsec;
    -      event.reset();
    --- End diff --
    
    Are you talking about re-ordering the `long elapsedTimeSinceRegularRunMsec = ` command and the `event.reset()` command? That shouldn't matter, I just moved the `if(eventTriggered)` because is seemed a bit clearer to put it close to the code that outputs `eventTriggered`.  The fix is the actual `if(eventTriggered)`.


---

[GitHub] giraph pull request #84: Fix BspServiceMaster bug

Posted by dlogothetis <gi...@git.apache.org>.
Github user dlogothetis commented on a diff in the pull request:

    https://github.com/apache/giraph/pull/84#discussion_r218507979
  
    --- Diff: giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java ---
    @@ -1379,9 +1379,15 @@ private boolean barrierOnWorkerList(String finishedWorkerPath,
     
           // Wait for a signal or timeout
           boolean eventTriggered = event.waitMsecs(eventLoopTimeout);
    +
    +      // If the event was triggered, we reset it. In the next loop run, we will
    +      // read ZK to get the new hosts.
    +      if (eventTriggered) {
    +        event.reset();
    +      }
    +
           long elapsedTimeSinceRegularRunMsec = System.currentTimeMillis() -
               lastRegularRunTimeMsec;
    -      event.reset();
    --- End diff --
    
    It's possible that after `event.waitMsecs` exits (due to timeout) and before `event.reset()` get called, the event get signaled. In this case, in the next loop `event.waitMsec` will timeout again and `logInfoOnlyRun` will continue to be false. 


---

[GitHub] giraph pull request #84: Fix BspServiceMaster bug

Posted by majakabiljo <gi...@git.apache.org>.
Github user majakabiljo commented on a diff in the pull request:

    https://github.com/apache/giraph/pull/84#discussion_r218551052
  
    --- Diff: giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java ---
    @@ -1379,9 +1379,15 @@ private boolean barrierOnWorkerList(String finishedWorkerPath,
     
           // Wait for a signal or timeout
           boolean eventTriggered = event.waitMsecs(eventLoopTimeout);
    +
    +      // If the event was triggered, we reset it. In the next loop run, we will
    +      // read ZK to get the new hosts.
    +      if (eventTriggered) {
    +        event.reset();
    +      }
    +
           long elapsedTimeSinceRegularRunMsec = System.currentTimeMillis() -
               lastRegularRunTimeMsec;
    -      event.reset();
    --- End diff --
    
    Making sure I understand what's happening, does reodering of commands here makes no difference and the fix is about only clearing event if it triggered?


---