You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Jacob Ferriero (Jira)" <ji...@apache.org> on 2020/04/30 20:40:00 UTC

[jira] [Updated] (BEAM-9856) HL7v2IO.ListHL7v2Messages should be refactored once Bulk Export API is available

     [ https://issues.apache.org/jira/browse/BEAM-9856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jacob Ferriero updated BEAM-9856:
---------------------------------
    Summary: HL7v2IO.ListHL7v2Messages should be refactored once Bulk Export API is available  (was: HL7v2IO.ListMessages should be refactored once Bulk Export API is available)

> HL7v2IO.ListHL7v2Messages should be refactored once Bulk Export API is available
> --------------------------------------------------------------------------------
>
>                 Key: BEAM-9856
>                 URL: https://issues.apache.org/jira/browse/BEAM-9856
>             Project: Beam
>          Issue Type: Improvement
>          Components: io-java-gcp
>            Reporter: Jacob Ferriero
>            Assignee: Jacob Ferriero
>            Priority: Major
>
> Currently the List Messages API paginates through in a single ProcessElement Call.
>  
> In the future if a bulk export API becomes available that would allow splitting on some dimension (e.g. create time), this should be refactored as a splittable DoFn or at least run several sub queries so that we are not listing an entire store in a single thread.
>  
> This could look like paginating through each hour of data w/ in the time frame that the store spans, in a separate thread.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)