[jira] [Created] (BEAM-9847) Verify If Triggering allows emitting eager results when processing a single element in HL7v2IO.

["Jacob Ferriero (Jira)" <ji...@apache.org>]

Jacob Ferriero created BEAM-9847:
------------------------------------

             Summary: Verify If Triggering allows emitting eager results when processing a single element in HL7v2IO.
                 Key: BEAM-9847
                 URL: https://issues.apache.org/jira/browse/BEAM-9847
             Project: Beam
          Issue Type: Task
          Components: io-java-gcp
            Reporter: Jacob Ferriero
            Assignee: Jacob Ferriero


Due to the nature of the HL7v2 API, HL7v2IO.ListHL7v2Messages follows the pattern of [paginating through all ListMessages results in a single ProcessElement call.|#diff-9cd595f078378218ccc01ce7e19ca766R447]]

 

Upon testing with customer against HL7v2 store with 350k messages  we observed that the ListMessages transform was not outputting any elements "hanging" for a long time (which was assumed to be the single thread paginating through all the results).

 
We added the following triggering in hopes that it would emit early results:

{code:java}
          .apply(
              Window.<HL7v2Message>into(new GlobalWindows())
                  .triggering(
                      AfterWatermark.pastEndOfWindow()
                          .withEarlyFirings(
                              AfterProcessingTime.pastFirstElementInPane()
                                  .plusDelayOf(Duration.standardSeconds(1))))
                  .discardingFiredPanes())
{code}


Our tests with this triggering seemed to indicate that it did not "hang" like the first test and seemed to output a more steady stream of elements.

Reviewer states that bundles must be committed atomically so no output elements of a (single process element call) can proceed to downstream stages until all output elements for that process element call are ready.

There may be other things at play here. Will seek to reproduce in a way that definitively confirms output elements can be eagerly output during the execution of a single process element call before it completes.

CC: [~pabloem]

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)