You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by GitBox <gi...@apache.org> on 2021/03/26 17:05:08 UTC

[GitHub] [samza] ajothomas opened a new pull request #1483: SAMZA-2641: Do not silently fail when InputOperator is not found in OperatorGraph for a SystemStream

ajothomas opened a new pull request #1483:
URL: https://github.com/apache/samza/pull/1483


   **Symptom**: 
   Samza job fails with a TaskCallbackTimeoutException if InputOperator is not found in OperatorGraph for a SystemStream
    
   **Cause:** 
   Currently, the pipeline silently fails when there is no InputOperator for a SystemStream in the OperatorGraph as the code doesn't handle such cases. The pipeline just silently fails and the job would throw an exception due to sync task callback timeout (TaskCallbackTimeoutException).
   
   **Changes:** 
   Call `failure` method of the TaskCallback with a SamzaException with the appropriate exception message.
    
   **Tests:** 
   Adding a unit test in TestStreamOperatorTask class to check if the appropriate exception was thrown for the use case discussed above.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [samza] mynameborat merged pull request #1483: SAMZA-2641: Do not silently fail when InputOperator is not found in OperatorGraph for a SystemStream

Posted by GitBox <gi...@apache.org>.
mynameborat merged pull request #1483:
URL: https://github.com/apache/samza/pull/1483


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [samza] mynameborat commented on a change in pull request #1483: SAMZA-2641: Do not silently fail when InputOperator is not found in OperatorGraph for a SystemStream

Posted by GitBox <gi...@apache.org>.
mynameborat commented on a change in pull request #1483:
URL: https://github.com/apache/samza/pull/1483#discussion_r602522926



##########
File path: samza-core/src/main/java/org/apache/samza/task/StreamOperatorTask.java
##########
@@ -146,6 +146,11 @@ public final void processAsync(IncomingMessageEnvelope ime, MessageCollector col
               callback.complete();
             }
           });
+        } else {
+          // If InputOperator is not found in the operator graph for a given SystemStream, throw an exception else the
+          // job will timeout due to async task callback timeout (TaskCallbackTimeoutException)
+          callback.failure(new SamzaException(
+              String.format("InputOperator not found in OperatorGraph for %s", systemStream.toString())));

Review comment:
       IIRC, this isn't a typical user related issue. The bug manifested due to non-determinism during generation of the DAG and got exposed due to a bug/change in beam transform names. Beam transform appended hashcode to the names of transform and as a result, the SSP associated for the input source known the framework end (message chooser and system consumers) differed from the SSP associated with the same input within `StreamOperatorTask`.
   
   Here is the example where it happened
   `2021-02-02 23:07:44.187 [main] LiSamzaConfigUtil [INFO] task.inputs system names parsed: [37-Combine_globally_Count_2_CreateVoid_Read_CreateSource__ParDo_BoundedSourceAsSDFWrapper__ParMultiDo_BoundedSourceAsSDFWrapper__output__PCollection_185544081_,`
   
   while the ApplicationDescriptor had a different view on the input name
   `2021-02-02 23:07:46.111 [main] ApplicationDescriptorImpl [INFO] Using NoOpSerde as the key serde for stream 37-Combine_globally_Count_2_CreateVoid_Read_CreateSource__ParDo_BoundedSourceAsSDFWrapper__ParMultiDo_BoundedSourceAsSDFWrapper__output__PCollection_1582906120_`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [samza] ajothomas commented on a change in pull request #1483: SAMZA-2641: Do not silently fail when InputOperator is not found in OperatorGraph for a SystemStream

Posted by GitBox <gi...@apache.org>.
ajothomas commented on a change in pull request #1483:
URL: https://github.com/apache/samza/pull/1483#discussion_r603592399



##########
File path: samza-core/src/test/java/org/apache/samza/task/TestStreamOperatorTask.java
##########
@@ -19,26 +19,30 @@
 
 package org.apache.samza.task;
 
+import java.lang.reflect.Field;
 import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
+import org.apache.samza.SamzaException;
 import org.apache.samza.context.Context;
 import org.apache.samza.context.JobContext;
 import org.apache.samza.operators.OperatorSpecGraph;
 import org.apache.samza.operators.impl.OperatorImplGraph;
 import org.apache.samza.system.IncomingMessageEnvelope;
+import org.apache.samza.system.SystemStream;
 import org.apache.samza.util.Clock;
 import org.junit.Test;
+import org.mockito.ArgumentCaptor;
 
-import static org.junit.Assert.fail;
+import static org.junit.Assert.*;

Review comment:
       Changed it. I just assumed that it was okay as there were no checkstyle checks on star imports during build and no build failures.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [samza] ajothomas commented on a change in pull request #1483: SAMZA-2641: Do not silently fail when InputOperator is not found in OperatorGraph for a SystemStream

Posted by GitBox <gi...@apache.org>.
ajothomas commented on a change in pull request #1483:
URL: https://github.com/apache/samza/pull/1483#discussion_r603592399



##########
File path: samza-core/src/test/java/org/apache/samza/task/TestStreamOperatorTask.java
##########
@@ -19,26 +19,30 @@
 
 package org.apache.samza.task;
 
+import java.lang.reflect.Field;
 import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
+import org.apache.samza.SamzaException;
 import org.apache.samza.context.Context;
 import org.apache.samza.context.JobContext;
 import org.apache.samza.operators.OperatorSpecGraph;
 import org.apache.samza.operators.impl.OperatorImplGraph;
 import org.apache.samza.system.IncomingMessageEnvelope;
+import org.apache.samza.system.SystemStream;
 import org.apache.samza.util.Clock;
 import org.junit.Test;
+import org.mockito.ArgumentCaptor;
 
-import static org.junit.Assert.fail;
+import static org.junit.Assert.*;

Review comment:
       Changed it. I just assumed that it was okay as there were no checkstyle checks on star imports during build.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [samza] mynameborat commented on pull request #1483: SAMZA-2641: Do not silently fail when InputOperator is not found in OperatorGraph for a SystemStream

Posted by GitBox <gi...@apache.org>.
mynameborat commented on pull request #1483:
URL: https://github.com/apache/samza/pull/1483#issuecomment-809829265


   @ajothomas can you look at the test failures that started with recent commits?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [samza] dxichen commented on pull request #1483: SAMZA-2641: Do not silently fail when InputOperator is not found in OperatorGraph for a SystemStream

Posted by GitBox <gi...@apache.org>.
dxichen commented on pull request #1483:
URL: https://github.com/apache/samza/pull/1483#issuecomment-810558441


   LGTM thanks for the fix!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [samza] dxichen commented on a change in pull request #1483: SAMZA-2641: Do not silently fail when InputOperator is not found in OperatorGraph for a SystemStream

Posted by GitBox <gi...@apache.org>.
dxichen commented on a change in pull request #1483:
URL: https://github.com/apache/samza/pull/1483#discussion_r602503011



##########
File path: samza-core/src/main/java/org/apache/samza/task/StreamOperatorTask.java
##########
@@ -146,6 +146,11 @@ public final void processAsync(IncomingMessageEnvelope ime, MessageCollector col
               callback.complete();
             }
           });
+        } else {
+          // If InputOperator is not found in the operator graph for a given SystemStream, throw an exception else the
+          // job will timeout due to async task callback timeout (TaskCallbackTimeoutException)
+          callback.failure(new SamzaException(
+              String.format("InputOperator not found in OperatorGraph for %s", systemStream.toString())));

Review comment:
       +1 @ajothomas this is exception is usually caused by missing config for the specific systemStream of the IncomingMessageEnvelope.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [samza] ajothomas commented on pull request #1483: SAMZA-2641: Do not silently fail when InputOperator is not found in OperatorGraph for a SystemStream

Posted by GitBox <gi...@apache.org>.
ajothomas commented on pull request #1483:
URL: https://github.com/apache/samza/pull/1483#issuecomment-810319147


   @mynameborat Pushed additional changes last evening. Travis build checks seem to work fine now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [samza] prateekm commented on a change in pull request #1483: SAMZA-2641: Do not silently fail when InputOperator is not found in OperatorGraph for a SystemStream

Posted by GitBox <gi...@apache.org>.
prateekm commented on a change in pull request #1483:
URL: https://github.com/apache/samza/pull/1483#discussion_r602469261



##########
File path: samza-core/src/main/java/org/apache/samza/task/StreamOperatorTask.java
##########
@@ -146,6 +146,11 @@ public final void processAsync(IncomingMessageEnvelope ime, MessageCollector col
               callback.complete();
             }
           });
+        } else {
+          // If InputOperator is not found in the operator graph for a given SystemStream, throw an exception else the
+          // job will timeout due to async task callback timeout (TaskCallbackTimeoutException)
+          callback.failure(new SamzaException(
+              String.format("InputOperator not found in OperatorGraph for %s", systemStream.toString())));

Review comment:
       Can we rephrase this message in terms users would understand (e.g. describe what they did wrong instead of the current state of the system)?. It'll also be useful to explain in the PR description when this can happen.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [samza] dxichen commented on a change in pull request #1483: SAMZA-2641: Do not silently fail when InputOperator is not found in OperatorGraph for a SystemStream

Posted by GitBox <gi...@apache.org>.
dxichen commented on a change in pull request #1483:
URL: https://github.com/apache/samza/pull/1483#discussion_r603649068



##########
File path: samza-core/src/main/java/org/apache/samza/task/StreamOperatorTask.java
##########
@@ -146,6 +147,13 @@ public final void processAsync(IncomingMessageEnvelope ime, MessageCollector col
               callback.complete();
             }
           });
+        } else {
+          // If InputOperator is not found in the operator graph for a given SystemStream, throw an exception else the
+          // job will timeout due to async task callback timeout (TaskCallbackTimeoutException)
+          final String errMessage = String.format("InputOperator not found in OperatorGraph for %s. Please check SystemStream "

Review comment:
       It would be helpful for debugging this in the case that the hash is different to log all available input operators to easily identify the mismatch using `operatorImplGraph#getAllInputOperators`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [samza] mynameborat commented on a change in pull request #1483: SAMZA-2641: Do not silently fail when InputOperator is not found in OperatorGraph for a SystemStream

Posted by GitBox <gi...@apache.org>.
mynameborat commented on a change in pull request #1483:
URL: https://github.com/apache/samza/pull/1483#discussion_r602463185



##########
File path: samza-core/src/test/java/org/apache/samza/task/TestStreamOperatorTask.java
##########
@@ -19,26 +19,30 @@
 
 package org.apache.samza.task;
 
+import java.lang.reflect.Field;
 import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
+import org.apache.samza.SamzaException;
 import org.apache.samza.context.Context;
 import org.apache.samza.context.JobContext;
 import org.apache.samza.operators.OperatorSpecGraph;
 import org.apache.samza.operators.impl.OperatorImplGraph;
 import org.apache.samza.system.IncomingMessageEnvelope;
+import org.apache.samza.system.SystemStream;
 import org.apache.samza.util.Clock;
 import org.junit.Test;
+import org.mockito.ArgumentCaptor;
 
-import static org.junit.Assert.fail;
+import static org.junit.Assert.*;

Review comment:
       There is mix and match of static star imports in the code. That said, we have settled on no star imports. Would you mind adding explicit imports instead? Maybe, clean the mockito ones in the file too :-)

##########
File path: samza-core/src/main/java/org/apache/samza/task/StreamOperatorTask.java
##########
@@ -146,6 +146,11 @@ public final void processAsync(IncomingMessageEnvelope ime, MessageCollector col
               callback.complete();
             }
           });
+        } else {
+          // If InputOperator is not found in the operator graph for a given SystemStream, throw an exception else the
+          // job will timeout due to async task callback timeout (TaskCallbackTimeoutException)
+          callback.failure(new SamzaException(
+              String.format("InputOperator not found in OperatorGraph for %s", systemStream.toString())));

Review comment:
       nit: `systemStream.toString()` seems redundant. passing `systemStream` should be sufficient.

##########
File path: samza-core/src/test/java/org/apache/samza/task/TestStreamOperatorTask.java
##########
@@ -80,4 +84,34 @@ public void testExceptionsInProcessInvokesTaskCallback() throws InterruptedExcep
         mockTaskCoordinator, mockTaskCallback);
     failureLatch.await();
   }
+
+  /**
+   * Tests if the appropriate SamzaException is propagated to the TaskCallback if there is no InputOperator for a given
+   * SystemStream in the OperatorGraph.
+   * */
+  @Test
+  public void testExceptionIfInputOperatorMissing() throws NoSuchFieldException, IllegalAccessException {
+    IncomingMessageEnvelope mockIme = mock(IncomingMessageEnvelope.class, RETURNS_DEEP_STUBS);
+    SystemStream testSystemStream = new SystemStream("foo", "bar");
+    when(mockIme.getSystemStreamPartition().getSystemStream()).thenReturn(testSystemStream);
+
+    OperatorImplGraph mockOperatorImplGraph = mock(OperatorImplGraph.class);
+    when(mockOperatorImplGraph.getInputOperator(anyObject())).thenReturn(null);
+    // OperatorImplGraph is a private field ('operatorImplGraph') in StreamOperatorTask and need to be set with
+    // the mocked value reflectively
+    StreamOperatorTask operatorTask = new StreamOperatorTask(mock(OperatorSpecGraph.class));
+    Field operatorImplGraphField = operatorTask.getClass().getDeclaredField("operatorImplGraph");
+    operatorImplGraphField.setAccessible(true);
+    operatorImplGraphField.set(operatorTask, mockOperatorImplGraph);

Review comment:
       would be simpler to add a setter to the field and make it `package private`? Tests can leverage this in future and don't have to repeat this boiler plate of reflection code.

##########
File path: samza-core/src/main/java/org/apache/samza/task/StreamOperatorTask.java
##########
@@ -146,6 +146,11 @@ public final void processAsync(IncomingMessageEnvelope ime, MessageCollector col
               callback.complete();
             }
           });
+        } else {
+          // If InputOperator is not found in the operator graph for a given SystemStream, throw an exception else the
+          // job will timeout due to async task callback timeout (TaskCallbackTimeoutException)
+          callback.failure(new SamzaException(

Review comment:
       Can we log the error as well similar to the pattern in the rest of the class before invoking `callback.failure(...)`? Callback failures are asynchronous and other exceptions can happen as well and hence, having a log helps drafting timelines during debugging.

##########
File path: samza-core/src/test/java/org/apache/samza/task/TestStreamOperatorTask.java
##########
@@ -80,4 +84,34 @@ public void testExceptionsInProcessInvokesTaskCallback() throws InterruptedExcep
         mockTaskCoordinator, mockTaskCallback);
     failureLatch.await();
   }
+
+  /**
+   * Tests if the appropriate SamzaException is propagated to the TaskCallback if there is no InputOperator for a given
+   * SystemStream in the OperatorGraph.
+   * */
+  @Test
+  public void testExceptionIfInputOperatorMissing() throws NoSuchFieldException, IllegalAccessException {
+    IncomingMessageEnvelope mockIme = mock(IncomingMessageEnvelope.class, RETURNS_DEEP_STUBS);
+    SystemStream testSystemStream = new SystemStream("foo", "bar");
+    when(mockIme.getSystemStreamPartition().getSystemStream()).thenReturn(testSystemStream);
+
+    OperatorImplGraph mockOperatorImplGraph = mock(OperatorImplGraph.class);
+    when(mockOperatorImplGraph.getInputOperator(anyObject())).thenReturn(null);
+    // OperatorImplGraph is a private field ('operatorImplGraph') in StreamOperatorTask and need to be set with
+    // the mocked value reflectively
+    StreamOperatorTask operatorTask = new StreamOperatorTask(mock(OperatorSpecGraph.class));
+    Field operatorImplGraphField = operatorTask.getClass().getDeclaredField("operatorImplGraph");
+    operatorImplGraphField.setAccessible(true);
+    operatorImplGraphField.set(operatorTask, mockOperatorImplGraph);
+
+    TaskCallback mockTaskCallback = mock(TaskCallback.class);
+    operatorTask.processAsync(mockIme, mock(MessageCollector.class), mock(TaskCoordinator.class), mockTaskCallback);
+    // ArgumentCaptor for capturing the SamzaException passed to the callback
+    ArgumentCaptor<Throwable> throwableCaptor = ArgumentCaptor.forClass(Throwable.class);
+    verify(mockTaskCallback, only()).failure(throwableCaptor.capture());
+    assertEquals(throwableCaptor.getValue().getClass(), SamzaException.class);
+    assertEquals(
+        throwableCaptor.getValue().getMessage(),
+        String.format("InputOperator not found in OperatorGraph for %s", testSystemStream.toString()));

Review comment:
       same as above. `testSystemStream` should be sufficient for the `String.format(...)` call.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [samza] ajothomas commented on a change in pull request #1483: SAMZA-2641: Do not silently fail when InputOperator is not found in OperatorGraph for a SystemStream

Posted by GitBox <gi...@apache.org>.
ajothomas commented on a change in pull request #1483:
URL: https://github.com/apache/samza/pull/1483#discussion_r603602715



##########
File path: samza-core/src/main/java/org/apache/samza/task/StreamOperatorTask.java
##########
@@ -146,6 +146,11 @@ public final void processAsync(IncomingMessageEnvelope ime, MessageCollector col
               callback.complete();
             }
           });
+        } else {
+          // If InputOperator is not found in the operator graph for a given SystemStream, throw an exception else the
+          // job will timeout due to async task callback timeout (TaskCallbackTimeoutException)
+          callback.failure(new SamzaException(
+              String.format("InputOperator not found in OperatorGraph for %s", systemStream.toString())));

Review comment:
       Added appropriate exceptions and corrected the description of the PR.

##########
File path: samza-core/src/main/java/org/apache/samza/task/StreamOperatorTask.java
##########
@@ -146,6 +146,11 @@ public final void processAsync(IncomingMessageEnvelope ime, MessageCollector col
               callback.complete();
             }
           });
+        } else {
+          // If InputOperator is not found in the operator graph for a given SystemStream, throw an exception else the
+          // job will timeout due to async task callback timeout (TaskCallbackTimeoutException)
+          callback.failure(new SamzaException(
+              String.format("InputOperator not found in OperatorGraph for %s", systemStream.toString())));

Review comment:
       Right, removed `toString()`

##########
File path: samza-core/src/test/java/org/apache/samza/task/TestStreamOperatorTask.java
##########
@@ -80,4 +84,34 @@ public void testExceptionsInProcessInvokesTaskCallback() throws InterruptedExcep
         mockTaskCoordinator, mockTaskCallback);
     failureLatch.await();
   }
+
+  /**
+   * Tests if the appropriate SamzaException is propagated to the TaskCallback if there is no InputOperator for a given
+   * SystemStream in the OperatorGraph.
+   * */
+  @Test
+  public void testExceptionIfInputOperatorMissing() throws NoSuchFieldException, IllegalAccessException {
+    IncomingMessageEnvelope mockIme = mock(IncomingMessageEnvelope.class, RETURNS_DEEP_STUBS);
+    SystemStream testSystemStream = new SystemStream("foo", "bar");
+    when(mockIme.getSystemStreamPartition().getSystemStream()).thenReturn(testSystemStream);
+
+    OperatorImplGraph mockOperatorImplGraph = mock(OperatorImplGraph.class);
+    when(mockOperatorImplGraph.getInputOperator(anyObject())).thenReturn(null);
+    // OperatorImplGraph is a private field ('operatorImplGraph') in StreamOperatorTask and need to be set with
+    // the mocked value reflectively
+    StreamOperatorTask operatorTask = new StreamOperatorTask(mock(OperatorSpecGraph.class));
+    Field operatorImplGraphField = operatorTask.getClass().getDeclaredField("operatorImplGraph");
+    operatorImplGraphField.setAccessible(true);
+    operatorImplGraphField.set(operatorTask, mockOperatorImplGraph);
+
+    TaskCallback mockTaskCallback = mock(TaskCallback.class);
+    operatorTask.processAsync(mockIme, mock(MessageCollector.class), mock(TaskCoordinator.class), mockTaskCallback);
+    // ArgumentCaptor for capturing the SamzaException passed to the callback
+    ArgumentCaptor<Throwable> throwableCaptor = ArgumentCaptor.forClass(Throwable.class);
+    verify(mockTaskCallback, only()).failure(throwableCaptor.capture());
+    assertEquals(throwableCaptor.getValue().getClass(), SamzaException.class);
+    assertEquals(
+        throwableCaptor.getValue().getMessage(),
+        String.format("InputOperator not found in OperatorGraph for %s", testSystemStream.toString()));

Review comment:
       Done.

##########
File path: samza-core/src/main/java/org/apache/samza/task/StreamOperatorTask.java
##########
@@ -146,6 +146,11 @@ public final void processAsync(IncomingMessageEnvelope ime, MessageCollector col
               callback.complete();
             }
           });
+        } else {
+          // If InputOperator is not found in the operator graph for a given SystemStream, throw an exception else the
+          // job will timeout due to async task callback timeout (TaskCallbackTimeoutException)
+          callback.failure(new SamzaException(

Review comment:
       Makes sense, done.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [samza] ajothomas commented on a change in pull request #1483: SAMZA-2641: Do not silently fail when InputOperator is not found in OperatorGraph for a SystemStream

Posted by GitBox <gi...@apache.org>.
ajothomas commented on a change in pull request #1483:
URL: https://github.com/apache/samza/pull/1483#discussion_r604343641



##########
File path: samza-core/src/main/java/org/apache/samza/task/StreamOperatorTask.java
##########
@@ -146,6 +147,13 @@ public final void processAsync(IncomingMessageEnvelope ime, MessageCollector col
               callback.complete();
             }
           });
+        } else {
+          // If InputOperator is not found in the operator graph for a given SystemStream, throw an exception else the
+          // job will timeout due to async task callback timeout (TaskCallbackTimeoutException)
+          final String errMessage = String.format("InputOperator not found in OperatorGraph for %s. Please check SystemStream "

Review comment:
       Done.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org