You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "ZihanLi58 (via GitHub)" <gi...@apache.org> on 2024/01/17 21:20:10 UTC

[PR] [GOBBLIN-1988] Fail the streaming container when OOM issue happens [gobblin]

ZihanLi58 opened a new pull request, #3862:
URL: https://github.com/apache/gobblin/pull/3862

   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
   
   
   ### JIRA
   - [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
       - https://issues.apache.org/jira/browse/GOBBLIN-1988
   
   
   ### Description
   - [ ] Here are some details about my PR, including screenshots (if applicable):
   Now when the OOM error happens, we just log out the error and keep the container running there without doing anything. In this case we should fail the container and retry the task.
   
   ### Tests
   - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
   Mock the OOM error and verified that before the change, task just hang there, and after the change, task got failed and retried as expected
   
   ### Commits
   - [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
       1. Subject is separated from body by a blank line
       2. Subject is limited to 50 characters
       3. Subject does not end with a period
       4. Subject uses the imperative mood ("add", not "adding")
       5. Body wraps at 72 characters
       6. Body explains "what" and "why", not "how"
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-1988] Fail the streaming container when OOM issue happens [gobblin]

Posted by "ZihanLi58 (via GitHub)" <gi...@apache.org>.
ZihanLi58 merged PR #3862:
URL: https://github.com/apache/gobblin/pull/3862


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-1988] Fail the streaming container when OOM issue happens [gobblin]

Posted by "ZihanLi58 (via GitHub)" <gi...@apache.org>.
ZihanLi58 commented on code in PR #3862:
URL: https://github.com/apache/gobblin/pull/3862#discussion_r1458164630


##########
gobblin-utility/src/main/java/org/apache/gobblin/util/LoggingUncaughtExceptionHandler.java:
##########
@@ -42,6 +45,7 @@ public LoggingUncaughtExceptionHandler(Optional<Logger> logger) {
 
   @Override
   public void uncaughtException(Thread t, Throwable e) {
+    exception = e;

Review Comment:
   it's being used in get method for the main thread to determine whether there is an exception in this thread: https://github.com/apache/gobblin/pull/3862/files#diff-5486722712e94a92c9e9129f7b3768c87cd7c3f3cf08613c24b9a5fd4131e6ebR163



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-1988] Fail the streaming container when OOM issue happens [gobblin]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3862:
URL: https://github.com/apache/gobblin/pull/3862#issuecomment-1896856631

   ## [Codecov](https://app.codecov.io/gh/apache/gobblin/pull/3862?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Comparison is base [(`694ed37`)](https://app.codecov.io/gh/apache/gobblin/commit/694ed37649ce4d48e4f4810287c7245339d19069?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 47.58% compared to head [(`0e35039`)](https://app.codecov.io/gh/apache/gobblin/pull/3862?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 46.01%.
   > Report is 18 commits behind head on master.
   
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3862      +/-   ##
   ============================================
   - Coverage     47.58%   46.01%   -1.58%     
   + Complexity    11050     2185    -8865     
   ============================================
     Files          2159      417    -1742     
     Lines         85429    18054   -67375     
     Branches       9493     2199    -7294     
   ============================================
   - Hits          40649     8307   -32342     
   + Misses        41080     8861   -32219     
   + Partials       3700      886    -2814     
   ```
   
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/gobblin/pull/3862?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-1988] Fail the streaming container when OOM issue happens [gobblin]

Posted by "AndyJiang99 (via GitHub)" <gi...@apache.org>.
AndyJiang99 commented on code in PR #3862:
URL: https://github.com/apache/gobblin/pull/3862#discussion_r1456712626


##########
gobblin-utility/src/main/java/org/apache/gobblin/util/LoggingUncaughtExceptionHandler.java:
##########
@@ -42,6 +45,7 @@ public LoggingUncaughtExceptionHandler(Optional<Logger> logger) {
 
   @Override
   public void uncaughtException(Thread t, Throwable e) {
+    exception = e;

Review Comment:
   The exception variable is set but never used, maybe missing something?



##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/StreamModelTaskRunner.java:
##########
@@ -155,13 +156,17 @@ protected void run() throws Exception {
     Thread thread = new Thread(() -> connectableStream.connect());
     thread.setName(this.getClass().getSimpleName());
     //Log uncaught exceptions (e.g.OOMEs) to prevent threads from dying silently
-    thread.setUncaughtExceptionHandler(new LoggingUncaughtExceptionHandler(Optional.absent()));
+    LoggingUncaughtExceptionHandler exceptionHandler = new LoggingUncaughtExceptionHandler(Optional.absent());
+    thread.setUncaughtExceptionHandler(exceptionHandler);
     thread.start();
-
-    if (!ExponentialBackoff.awaitCondition().callable(() -> this.forks.keySet().stream().map(Optional::get).allMatch(Fork::isDone)).
+    if (!ExponentialBackoff.awaitCondition().callable(() ->
+        this.forks.keySet().stream().map(Optional::get).allMatch(Fork::isDone) || exceptionHandler.getException() != null).
         initialDelay(initialDelay).maxDelay(initialDelay).maxWait(TimeUnit.MINUTES.toMillis(maxWaitInMinute)).await()) {
       throw new TimeoutException("Forks did not finish withing specified timeout.");
     }
+    if (exceptionHandler.getException() != null) {
+      throw Throwables.propagate(exceptionHandler.getException());
+    }

Review Comment:
   You mentioned a mock to test this functionality to fail the container and retry the job again, but I don't see the rests associated with that



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org