[jira] [Comment Edited] (BEAM-646) Get runners out of the apply()

["Ning Kang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16953250#comment-16953250 ] 

Ning Kang edited comment on BEAM-646 at 10/16/19 10:51 PM:
-----------------------------------------------------------

Hi,

This is Ning. I'm working on a project that supports InteractiveRunner (Python3+) when users use Beam in notebook environment.

Would apply() interception in runner still be useful as a hook for non-Beam related but interactivity related features such as collecting IPython/notebook cell information when a PTransfrom is applied?

Of course, we can also have all pipelines support interactivity by making the interception inside pipelines themselves. But it's unlikely that all runners would/could take a pipeline with interactivity logic at this moment. And those ipython/notebook dependencies probably shouldn't be introduced into pipeline itself.

Would there be any APIs that support invoking arbitrary external logic at different stages of building a pipeline when pipeline is completely decoupled from runner?

Thanks!


was (Author: ningk):
Hi,

This is Ning. I'm working a project that supports InteractiveRunner (Python3+) when users use Beam in notebook environment.

Would apply() interception in runner still be useful as a hook for non-Beam related but interactivity related features such as collecting IPython/notebook cell information when a PTransfrom is applied?

Of course, we can also have all pipelines support interactivity by making the interception inside pipelines themselves. But it's unlikely that all runners would/could take a pipeline with interactivity logic at this moment. And those ipython/notebook dependencies probably shouldn't be introduced into pipeline itself.

Would there be any APIs that support invoking arbitrary external logic at different stages of building a pipeline when pipeline is completely decoupled from runner?

Thanks!

> Get runners out of the apply()
> ------------------------------
>
>                 Key: BEAM-646
>                 URL: https://issues.apache.org/jira/browse/BEAM-646
>             Project: Beam
>          Issue Type: Improvement
>          Components: beam-model, sdk-java-core
>            Reporter: Kenneth Knowles
>            Assignee: Thomas Groh
>            Priority: Major
>              Labels: backwards-incompatible
>             Fix For: 0.6.0
>
>
> Right now, the runner intercepts calls to apply() and replaces transforms as we go. This means that there is no "original" user graph. For portability and misc architectural benefits, we would like to build the original graph first, and have the runner override later.
> Some runners already work in this manner, but we could integrate it more smoothly, with more validation, via some handy APIs on e.g. the Pipeline object.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)