[jira] [Closed] (BEAM-65) SplittableDoFn

["Eugene Kirpichov (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eugene Kirpichov closed BEAM-65.
--------------------------------
       Resolution: Fixed
    Fix Version/s: 2.2.0

SDF has been available in the Beam model and implemented in Direct and Dataflow streaming runner for a while. Other work can be tracked separately - there is no clear completion criterion for the current Jira justifying keeping it open.

> SplittableDoFn
> --------------
>
>                 Key: BEAM-65
>                 URL: https://issues.apache.org/jira/browse/BEAM-65
>             Project: Beam
>          Issue Type: New Feature
>          Components: beam-model
>            Reporter: Daniel Halperin
>            Assignee: Eugene Kirpichov
>            Priority: Minor
>             Fix For: 2.2.0
>
>
> SplittableDoFn is a proposed enhancement for "dynamically splittable work" to the Beam model.
> Among other things, it would allow a unified implementation of bounded/unbounded sources with dynamic work rebalancing and the ability to express multiple scalable steps (e.g., global expansion -> file sizing & parsing -> splitting files into independently-processable blocks) via composition rather than inheritance.
> This would make it much easier to implement many types of sources, to modify and reuse existing sources. Also, it would improve scalability of the Beam model by moving things like splitting a source from the control plane (where it is today -- glob -> List<FileBasedSource> sent over service APIs) into the data plane (PCollection<Glob> -> PCollection<FileName> -> ...).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)