[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #3214: [GOBBLIN-1359] Fix the error that Exception thrown by writer be swallowed by Reactivex and cause jobs to hang

[GitBox <gi...@apache.org>]

autumnust commented on a change in pull request #3214:
URL: https://github.com/apache/incubator-gobblin/pull/3214#discussion_r565719362



##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/StreamModelTaskRunner.java
##########
@@ -86,7 +87,10 @@ protected void run() throws Exception {
 
     // The cancel is not propagated to the extractor's record generator when it has been turned into a hot Flowable
     // by publish, so set the shutdownRequested flag on cancel to stop the extractor
-    Flowable streamWithShutdownOnCancel = connectableStream.doOnCancel(() -> this.shutdownRequested.set(true));
+    Flowable streamWithShutdownOnCancel = connectableStream.doOnCancel(() -> {
+      this.shutdownRequested.set(true);
+      this.extractor.shutdown();

Review comment:
       I don't quite remember how was the failure propagated when the slow write happens, but feels like the right fix here is to add finally block in the place where `extractor.shutdown()` is called in `Task` (e.g. line 514) after catching failure, instead of forcing extractor closure here. 

##########
File path: gobblin-core-base/src/main/java/org/apache/gobblin/source/extractor/extract/FlushingExtractor.java
##########
@@ -298,6 +298,14 @@ protected void publishTaskOutput() throws IOException {
     this.flushPublisher.get().publish(Collections.singletonList(workUnitState));
   }
 
+  @Override
+  public void shutdown() {

Review comment:
       What's the reason super.shutdown is not called, is this intentional ? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org