[jira] [Commented] (APEXCORE-599) Data not fully processed when operator terminates via ShutdownException

["Thomas Weise (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/APEXCORE-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798979#comment-15798979 ] 

Thomas Weise commented on APEXCORE-599:
---------------------------------------

I could reproduce it on a Ubuntu VM by picking a specific test and run it long enough like this:

{code}
while mvn test -Dtest=ParDoBoundTranslatorTest#testMultiOutputParDoWithSideInputs; do :; done
{code}

Enable tuple tracing in the pipeline options and it will make it easier to see how the last tuple even though it was emitted by the input operator in the streaming window *before* shutdown never makes it downstream.



> Data not fully processed when operator terminates via ShutdownException
> -----------------------------------------------------------------------
>
>                 Key: APEXCORE-599
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-599
>             Project: Apache Apex Core
>          Issue Type: Bug
>    Affects Versions: 3.5.0
>            Reporter: Thomas Weise
>
> Observed in the form of flaky unit tests in Beam. The pipelines terminate using shutdown by the operator when there is no more input and occasionally the test will fail because the cluster exits without having processed fully. The bandaid is to have a delay before shutdown is issued, allowing the last tuples to be pushed downstream. See BEAM-1140 for details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)