[jira] [Resolved] (BEAM-9847) Verify If Triggering allows emitting eager results when processing a single element in HL7v2IO.

["Jacob Ferriero (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jacob Ferriero resolved BEAM-9847.
----------------------------------
    Fix Version/s: Not applicable
       Resolution: Not A Bug

> Verify If Triggering allows emitting eager results when processing a single element in HL7v2IO.
> -----------------------------------------------------------------------------------------------
>
>                 Key: BEAM-9847
>                 URL: https://issues.apache.org/jira/browse/BEAM-9847
>             Project: Beam
>          Issue Type: Task
>          Components: io-java-gcp
>            Reporter: Jacob Ferriero
>            Assignee: Jacob Ferriero
>            Priority: Minor
>             Fix For: Not applicable
>
>
> Due to the nature of the HL7v2 API, HL7v2IO.ListHL7v2Messages follows the pattern of [paginating through all ListMessages results in a single ProcessElement call.|#diff-9cd595f078378218ccc01ce7e19ca766R447]]
>  
> Upon testing with customer against HL7v2 store with 350k messages  we observed that the ListMessages transform was not outputting any elements "hanging" for a long time (which was assumed to be the single thread paginating through all the results).
>  
> We added the following triggering in hopes that it would emit early results:
> {code:java}
>           .apply(
>               Window.<HL7v2Message>into(new GlobalWindows())
>                   .triggering(
>                       AfterWatermark.pastEndOfWindow()
>                           .withEarlyFirings(
>                               AfterProcessingTime.pastFirstElementInPane()
>                                   .plusDelayOf(Duration.standardSeconds(1))))
>                   .discardingFiredPanes())
> {code}
> Our tests with this triggering seemed to indicate that it did not "hang" like the first test and seemed to output a more steady stream of elements.
> Reviewer states that bundles must be committed atomically so no output elements of a (single process element call) can proceed to downstream stages until all output elements for that process element call are ready.
> There may be other things at play here. Will seek to reproduce in a way that definitively confirms output elements can be eagerly output during the execution of a single process element call before it completes.
> CC: [~pabloem]
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)