You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Tianyang Hu (Jira)" <ji...@apache.org> on 2022/01/10 19:19:00 UTC
[jira] [Updated] (BEAM-13626) Outdated Transform.Inputs values when chaining expanded transforms
[ https://issues.apache.org/jira/browse/BEAM-13626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tianyang Hu updated BEAM-13626:
-------------------------------
Description:
For:
pcol := ...
tmp := beam.CrossLanguage(..., {"foo": pcol}, ...}
out := beam.CrossLanguage(..., {"bar": tmp}, ...}
In the result model pipeline, the second expanded transform's inputs may hold a map from local name to the original PCollection/node ID, even if the first expansion has changed the output IDs.
Looks like a fix is to move the output name remapping after mergeExpandedWithPipeline: https://github.com/apache/beam/blob/928da54e1a95219bfbfa420fbf9c8fb5514fd683/sdks/go/pkg/beam/core/runtime/graphx/translate.go#L160-L163
was:
For:
pcol := ...
tmp := beam.CrossLanguage(..., map[string]PCollection{"foo": pcol}, ...}
out := beam.CrossLanguage(..., map[string]PCollection{"bar": tmp}, ...}
In the result model pipeline, the second expanded transform's inputs may hold a map from local name to the original PCollection/node ID, even if the first expansion has changed the output IDs.
Looks like a fix is to move the output name remapping after mergeExpandedWithPipeline: https://github.com/apache/beam/blob/928da54e1a95219bfbfa420fbf9c8fb5514fd683/sdks/go/pkg/beam/core/runtime/graphx/translate.go#L160-L163
> Outdated Transform.Inputs values when chaining expanded transforms
> ------------------------------------------------------------------
>
> Key: BEAM-13626
> URL: https://issues.apache.org/jira/browse/BEAM-13626
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
> Reporter: Tianyang Hu
> Assignee: Tianyang Hu
> Priority: P2
>
> For:
> pcol := ...
> tmp := beam.CrossLanguage(..., {"foo": pcol}, ...}
> out := beam.CrossLanguage(..., {"bar": tmp}, ...}
> In the result model pipeline, the second expanded transform's inputs may hold a map from local name to the original PCollection/node ID, even if the first expansion has changed the output IDs.
> Looks like a fix is to move the output name remapping after mergeExpandedWithPipeline: https://github.com/apache/beam/blob/928da54e1a95219bfbfa420fbf9c8fb5514fd683/sdks/go/pkg/beam/core/runtime/graphx/translate.go#L160-L163
--
This message was sent by Atlassian Jira
(v8.20.1#820001)