[jira] [Comment Edited] (BEAM-8418) Fix handling of Impulse transform in Dataflow runner.

["Valentyn Tymofieiev (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16953319#comment-16953319 ] 

Valentyn Tymofieiev edited comment on BEAM-8418 at 10/17/19 2:12 AM:
---------------------------------------------------------------------

[~robertwb] do you have an insight why URL-escaping was chosen to encode impulse payload in [1], and advice on how the encoding should be reconciled here?

Thanks. 

[1] https://github.com/apache/beam/pull/6581/files#diff-28cee3232c08db90a95ec550f0b182ccR517


was (Author: tvalentyn):
[~robertwb] do you have an insight why URL-escaping was chosen, and advise on how the encoding should be reconciled here?

Thanks. 

> Fix handling of Impulse transform in Dataflow runner. 
> ------------------------------------------------------
>
>                 Key: BEAM-8418
>                 URL: https://issues.apache.org/jira/browse/BEAM-8418
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py-core
>            Reporter: Valentyn Tymofieiev
>            Priority: Major
>
> Following pipeline fails on Dataflow runner unless we use beam_fn_api experiment.
> {noformat}
> class NoOpDoFn(beam.DoFn):
>   def process(self, element):
>     return element
> p = beam.Pipeline(options=pipeline_options)
> _ = p | beam.Impulse() | beam.ParDo(NoOpDoFn())
> result = p.run()
> {noformat}
> The reason is that we encode Impluse payload using url-escaping in [1], while Dataflow runner expects base64 encoding in non-fnapi mode. In FnApi mode, DF runner expects URL escaping.
> We should fix or otherwise the encoding in non-FnAPI path, and add a ValidatesRunner test that catches this error.   
> [1] https://github.com/apache/beam/blob/12d07745835e1b9c1e824b83beeeadf63ab4b234/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L633



--
This message was sent by Atlassian Jira
(v8.3.4#803005)