You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Robert Burke (JIRA)" <ji...@apache.org> on 2019/07/09 00:05:00 UTC
[jira] [Work started] (BEAM-7709) Flattening multiple outputs of a
ParDoN fails
[ https://issues.apache.org/jira/browse/BEAM-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on BEAM-7709 started by Robert Burke.
------------------------------------------
> Flattening multiple outputs of a ParDoN fails
> ---------------------------------------------
>
> Key: BEAM-7709
> URL: https://issues.apache.org/jira/browse/BEAM-7709
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
> Affects Versions: Not applicable
> Reporter: Robert Burke
> Assignee: Robert Burke
> Priority: Major
> Fix For: Not applicable
>
>
> If a user does a beam.ParDoN for pardo > 2 and then passes one or more of the outputs to a flatten, then if the flatten occurs SDK side, it currently creates multiple flatten nodes, which then triggers the downstream pardo (the DoFn that consumes the Flatten's output) to be initialized multiple times for a single bundle.
> The fix is to pre-emptively populate the input links with the first created flatten, so subsequent tracings of the plan use the same flatten node the same way the Go direct runner does[1]. That would happen in the exec translate code.
> [[1] https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/direct/direct.go#L299|https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/direct/direct.go#L299]
> [[2] https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/translate.go#L493|https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/translate.go#L493]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)