You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Luke Cwik (Jira)" <ji...@apache.org> on 2020/04/23 20:05:00 UTC

[jira] [Commented] (BEAM-7199) Better optimize Portable pipelines

    [ https://issues.apache.org/jira/browse/BEAM-7199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090888#comment-17090888 ] 

Luke Cwik commented on BEAM-7199:
---------------------------------

The optimizations that Python does should really be moved out of the Python SDK and implemented on the runner/job server side.

This was done for splittable DoFns but other optimizations such as combiner lifting are still needed.

> Better optimize Portable pipelines
> ----------------------------------
>
>                 Key: BEAM-7199
>                 URL: https://issues.apache.org/jira/browse/BEAM-7199
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-py-core
>            Reporter: Ankur Goenka
>            Priority: Major
>              Labels: portability
>
> Python has an experimental flag pre_optimize=all which does pre-optimization of python pipelines by fusing operators.
> Python optimization is expected to be better than the one in java because it has more information about the pipeline.
> Make java pipeline optimization at par with python so that the benefits can be shared by all languages.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)