[jira] [Reopened] (BEAM-10708) InteractiveRunner cannot execute pipeline with cross-language transform

["Brian Hulette (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-10708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brian Hulette reopened BEAM-10708:
----------------------------------
    Assignee:     (was: Ning)

I don't think we should close this. The beam_sql magic is a great workaround for a specific (important) use-case, but we should still track the underlying issue. There are many other external transforms and it would be nice for them all to work with the InteractiveRunner.

It would also be nice to fail more gracefully as an intermediate step.

> InteractiveRunner cannot execute pipeline with cross-language transform
> -----------------------------------------------------------------------
>
>                 Key: BEAM-10708
>                 URL: https://issues.apache.org/jira/browse/BEAM-10708
>             Project: Beam
>          Issue Type: Bug
>          Components: cross-language
>            Reporter: Brian Hulette
>            Priority: P2
>             Fix For: Missing
>
>          Time Spent: 49h
>  Remaining Estimate: 0h
>
> The InteractiveRunner crashes when given a pipeline that includes a cross-language transform.
> Here's the example I tried to run in a jupyter notebook:
> {code:python}
> p = beam.Pipeline(InteractiveRunner())
> pc = (p | SqlTransform("""SELECT
>             CAST(1 AS INT) AS `id`,
>             CAST('foo' AS VARCHAR) AS `str`,
>             CAST(3.14  AS DOUBLE) AS `flt`"""))
> df = interactive_beam.collect(pc)
> {code}
> The problem occurs when [pipeline_fragment.py|https://github.com/apache/beam/blob/dce1eb83b8d5137c56ac58568820c24bd8fda526/sdks/python/apache_beam/runners/interactive/pipeline_fragment.py#L66] creates a copy of the pipeline by [writing it to proto and reading it back|https://github.com/apache/beam/blob/dce1eb83b8d5137c56ac58568820c24bd8fda526/sdks/python/apache_beam/runners/interactive/pipeline_fragment.py#L120]. Reading it back fails because some of the pipeline is not written in Python.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)