[jira] [Closed] (FLINK-14709) Allow outputting elements in close method of chained drivers.

["Aljoscha Krettek (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-14709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aljoscha Krettek closed FLINK-14709.
------------------------------------
    Fix Version/s: 1.10.0
         Assignee: David Moravek
       Resolution: Fixed

Merged on master in e583a1c251038e9b9606304b0a4be6a5b1a268de and following commits.

> Allow outputting elements in close method of chained drivers.
> -------------------------------------------------------------
>
>                 Key: FLINK-14709
>                 URL: https://issues.apache.org/jira/browse/FLINK-14709
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Task
>    Affects Versions: 1.7.2, 1.8.1, 1.9.1
>            Reporter: David Moravek
>            Assignee: David Moravek
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.10.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently, BatchTask and DataSourceTask only allow outputting elements in close method of "rich" operators, that they directly execute.
> Task workflow is as follows:
> 1) open "head" driver  (calls "open" method on udf)
> 2) open chained drivers
> 3) run "head" driver
> 4) close "head" driver (calls "close" method on udf)
> 5) close output collector (no elements can be collected after this point)
> 6) close chained drivers
> In order to properly support outputs from close method, we want to switch 6) and 5). We also need to tweak implementation of Reduce / Combine chained drivers, because they dispose sorters in closeTask method (this should be done in the close method).
> This would bring huge performance improvement for Beam users, because we could properly implement bundling on batch (whole partition = single bundle).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)