You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Luke Cwik (JIRA)" <ji...@apache.org> on 2019/08/05 19:48:00 UTC
[jira] [Comment Edited] (BEAM-7850) Make Environment a top level
attribute of PTransform
[ https://issues.apache.org/jira/browse/BEAM-7850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900366#comment-16900366 ]
Luke Cwik edited comment on BEAM-7850 at 8/5/19 7:47 PM:
---------------------------------------------------------
SdkFunctionSpec = FuctionSpec + environment
I would suggest getting rid of SdkFunctionSpec completely in favor of FunctionSpec and add the environment id explicitly into the WindowingStrategy
I migrated Coder to use FunctionSpec instead of SdkFunctionSpec since coders are implicitly understood by all neighboring PTransforms in BEAM-3204.
Eventually in the future we may want to have SdkFunctionSpec appear again if we ever support first class functions outside of the context of a PTransform.
was (Author: lcwik):
SdkFunctionSpec = FuctionSpec + environment
I would suggest getting rid of SdkFunctionSpec completely in favor of FunctionSpec and add the environment id explicitly into the WindowingStrategy
I migrated Coder to use FunctionSpec instead of SdkFunctionSpec since coders are implicitly understood by all neighboring PTransforms in BEAM-3204.
Consider making this a subtask under BEAM-3221.
Eventually in the future we may want to have SdkFunctionSpec appear again if we ever support first class functions outside of the context of a PTransform.
> Make Environment a top level attribute of PTransform
> ----------------------------------------------------
>
> Key: BEAM-7850
> URL: https://issues.apache.org/jira/browse/BEAM-7850
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
> Reporter: Chamikara Jayalath
> Priority: Major
>
> Currently Environment is not a top level attribute of the PTransform (of runner API proto).
> [https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L99]
> Instead it is hidden inside various payload objects. For example, for ParDo, environment will be inside SdkFunctionSpec of ParDoPayload.
> [https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L99]
>
> This makes tracking environment of different types of PTransforms harder and we have to fork code (on the type of PTransform) to extract the Environment where the PTransform should be executed. It will probably be simpler to just make Environment a top level attribute of PTransform.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)