[jira] [Issue Comment Deleted] (BEAM-702) Simple pattern for per-bundle and per-DoFn Closeable resources

["Pei He (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pei He updated BEAM-702:
------------------------
    Comment: was deleted

(was: 1. If there are failures in one bundle, we can easily call teardown()s of DoFns associated with this bundle. (I think we had plan to do it, I am not sure it has been done or not.)
The tricky part is that if there are failures in other PTransforms or in the environment while we are processing the bundles. Before runners fail the whole job, we want runners be able to call teardown() for all running ParDos and their bundles in processing. We don't have a plan for this part yet.

2. We should document and clarify whether finishBundle() will be called or not if the bundle fails.)

> Simple pattern for per-bundle and per-DoFn Closeable resources
> --------------------------------------------------------------
>
>                 Key: BEAM-702
>                 URL: https://issues.apache.org/jira/browse/BEAM-702
>             Project: Beam
>          Issue Type: Improvement
>            Reporter: Eugene Kirpichov
>
> Dealing with Closeable resources inside a processElement call is easy: simply use try-with-resources.
> However, bundle- or DoFn-scoped resources, such as long-lived database connections, are less convenient to deal with: you have to open them in startBundle and conditionally close in finishBundle (likewise setup/teardown), taking special care if there's multiple resources to close all of them.
> Perhaps we should provide something like Guava's Closer to DoFn's https://github.com/google/guava/wiki/ClosingResourcesExplained. Ideally, the user would need to only write a startBundle() or setup() method, but not write finishBundle() or teardown() - resources would be closed automatically.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)