You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Kenneth Knowles (JIRA)" <ji...@apache.org> on 2016/10/23 22:02:58 UTC
[jira] [Created] (BEAM-801) SplittableParDo must be a
pseudo-primitive, not ParDo(OldDoFn)
Kenneth Knowles created BEAM-801:
------------------------------------
Summary: SplittableParDo must be a pseudo-primitive, not ParDo(OldDoFn)
Key: BEAM-801
URL: https://issues.apache.org/jira/browse/BEAM-801
Project: Beam
Issue Type: Bug
Components: runner-core, runner-direct
Reporter: Kenneth Knowles
Today, runners-core contains an expansion of {{ParDo(DoFn)}} that implements it via unsupported features of {{ParDo(OldDoFn)}}. This expansion is used by the DirectRunner.
The right approach to provide a ready implementation in runners-core is the one taken by {{GroupAlsoByWindowViaWindowSetDoFn}} where the unsupported features are provided to the constructor of the {{DoFn}}, rather than expected to be passed through the {{ProcessContext}}.
These features are not guaranteed to be part of the public state & timers API of {{DoFn}}, particularly in the case of watermark holds, so it is not prudent to plan on waiting for the new state & timers API and porting over.
These issues create real blockers by causing dependency cycles between implementing {{DoFn}} fully (requires no use of unsupported features), implementing new state and timers (requires new {{DoFn}}), and keeping the hack until new {{DoFn}} has state and timers (requires use of unsupported features).
To break the cycle, the tests that rely on unsupported features are being disabled. They can be re-enabled either by following the design suggested above (probably the most robust approach) or by waiting and porting to new features as they are available.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)