You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Charles Chen (JIRA)" <ji...@apache.org> on 2018/06/27 00:10:00 UTC
[jira] [Resolved] (BEAM-3512) Python PTransform overrides do not
completely remove the overriden transform
[ https://issues.apache.org/jira/browse/BEAM-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Charles Chen resolved BEAM-3512.
--------------------------------
Resolution: Fixed
Fix Version/s: 2.3.0
> Python PTransform overrides do not completely remove the overriden transform
> ----------------------------------------------------------------------------
>
> Key: BEAM-3512
> URL: https://issues.apache.org/jira/browse/BEAM-3512
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Affects Versions: 2.2.0
> Reporter: Charles Chen
> Assignee: Charles Chen
> Priority: Major
> Fix For: 2.3.0
>
>
> In Python Beam runners, we support the use of PTransformOverrides to allow runners to override the behavior of specific transforms. The overriding mechanism seeks to excise the original transform (with its composite children transforms, if any) from the original graph, and graft the new replacement to any inputs and outputs of the original transform. However, this mechanism does not completely remove these pieces. Specifically:
> 1. Composite transform parts are not removed from the overridden AppliedPTransform; we only attempt to remove all the labels.
> 2. Not all labels are recursively removed; there is a bug in the label removing logic so that only the labels on every other level of the nested composite PTransform hierarchy are removed (see [https://github.com/apache/beam/blob/61777b4338733d99f4f858be0d7d0313ec138a06/sdks/python/apache_beam/pipeline.py#L171]).
> 3. Some excised parts of the pipeline still seem to be run: if we set "part.transform = None" for each overridden composite part of the pipeline, some tests fail with an error indicating that it could not execute the "None" transform. This implies that these parts are still somehow present in the execution graph.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)