You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Kenneth Knowles (Jira)" <ji...@apache.org> on 2021/01/19 23:59:00 UTC
[jira] [Commented] (BEAM-7850) Make Environment a top level
attribute of PTransform
[ https://issues.apache.org/jira/browse/BEAM-7850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268286#comment-17268286 ]
Kenneth Knowles commented on BEAM-7850:
---------------------------------------
I just found this because I stumbled across the change. Sorry to bring up a really old Jira.
Since almost all PTransforms are composites, what is the spec for how the environment on a composite relates to environment on subtransforms? I guess you expect them all to be the same?
The reason it came up is the use of annotations for both privacy properties and resource hints. Resource hints don't really make sense on a composite. But we had been saying that they should go on environment instead. But then environments got moved to composites so the whole discussion is confusing.
> Make Environment a top level attribute of PTransform
> ----------------------------------------------------
>
> Key: BEAM-7850
> URL: https://issues.apache.org/jira/browse/BEAM-7850
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
> Reporter: Chamikara Madhusanka Jayalath
> Assignee: Chamikara Madhusanka Jayalath
> Priority: P2
> Fix For: 2.19.0
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Currently Environment is not a top level attribute of the PTransform (of runner API proto).
> [https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L99]
> Instead it is hidden inside various payload objects. For example, for ParDo, environment will be inside SdkFunctionSpec of ParDoPayload.
> [https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L99]
>
> This makes tracking environment of different types of PTransforms harder and we have to fork code (on the type of PTransform) to extract the Environment where the PTransform should be executed. It will probably be simpler to just make Environment a top level attribute of PTransform.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)