You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Robert Burke (JIRA)" <ji...@apache.org> on 2018/07/11 17:32:00 UTC
[jira] [Commented] (BEAM-4531) [Go SDK] Allow Dynamic Structural
DoFns
[ https://issues.apache.org/jira/browse/BEAM-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540437#comment-16540437 ]
Robert Burke commented on BEAM-4531:
------------------------------------
I determined that BEAM-4765 would be a more fruitful performance fix, so I've unassigned myself from this task. Once I've ushered my current outstanding tasks into the repo, then I'll look at this again, if it's still available.
> [Go SDK] Allow Dynamic Structural DoFns
> ---------------------------------------
>
> Key: BEAM-4531
> URL: https://issues.apache.org/jira/browse/BEAM-4531
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
> Reporter: Robert Burke
> Priority: Major
>
> Presently, the Go SDK permits 3 kinds of functions for use in ParDos.
> # Native functions
> # Structural DoFns with correctly named methods, that reflect the bundle lifecycle, or combine lifecycle (Setup StartBundle, FinishBundle, ProcessElement, Teardown etc.) These also permit limited Stateful ParDos, such as for caching expensive network responses for re-use when processing state.
> # Dynamic functions (DynFns) which permit the use of closured global state, and dynamic input types which are generated at pipeline runtime.
> There's presently no way to generate a stateful function that relies on closured global state and needs to be aware of the bundle lifecycle in the worker harness. In short, there's no way to create a Dynamic Structural DoFn.
> To implement this, in particular, [graph.DynFn |http://example.com](https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/graph/fn.go#L70)
> will need to be modified so that multiple methods can be returned, likely as methods on a struct, so that [graph.NewFn](https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/graph/fn.go#L81) can populate the [graph.Fn.methods](https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/graph/fn.go#L40) map correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)