You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Eugene Kirpichov (JIRA)" <ji...@apache.org> on 2017/03/28 23:43:41 UTC
[jira] [Created] (BEAM-1824) Adapter for running SDF on a
statically known input as a BoundedSource
Eugene Kirpichov created BEAM-1824:
--------------------------------------
Summary: Adapter for running SDF on a statically known input as a BoundedSource
Key: BEAM-1824
URL: https://issues.apache.org/jira/browse/BEAM-1824
Project: Beam
Issue Type: New Feature
Components: runner-dataflow, sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Eugene Kirpichov
[~bchambers] suggested the following idea: while the runner implementation of SDF [BEAM-65] is not yet complete enough to support dynamic rebalancing (especially over the Fn API), we can special-case the case of Create.of(single input) + ParDo(SDF) by running it via BoundedSource.
This will allow us to start transitioning bounded IO connectors to SDF API while preserving the dynamic rebalancing feature in the common case when the source is known at pipeline submission time.
And then, when SDF runner support catches up, we'll simply add APIs to the IO connectors for reading from a PCollection of inputs, and those will enjoy the same benefits. Actually we can add such APIs earlier, with the caveat that they won't support dynamic rebalancing, but in this case it's ok because there'll be no performance regression because these APIs didn't exist before.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)