[jira] [Created] (BEAM-8418) Fix handling of Impulse transform in Dataflow runner.

["Valentyn Tymofieiev (Jira)" <ji...@apache.org>]

Valentyn Tymofieiev created BEAM-8418:
-----------------------------------------

             Summary: Fix handling of Impulse transform in Dataflow runner. 
                 Key: BEAM-8418
                 URL: https://issues.apache.org/jira/browse/BEAM-8418
             Project: Beam
          Issue Type: Bug
          Components: sdk-py-core
            Reporter: Valentyn Tymofieiev


Following pipeline fails on Dataflow runner unless we use beam_fn_api experiment.
{noformat}
class NoOpDoFn(beam.DoFn):
  def process(self, element):
    return element

p = beam.Pipeline(options=pipeline_options)
_ = p | beam.Impulse() | beam.ParDo(NoOpDoFn())
result = p.run()
{noformat}

The reason is that we encode Impluse payload using url-escaping in [1], while Dataflow runner expects base64 encoding in non-fnapi mode. In FnApi mode, DF runner expects URL escaping.

We should fix or otherwise the encoding in non-FnAPI path, and add a ValidatesRunner test that catches this error.   

[1] https://github.com/apache/beam/blob/12d07745835e1b9c1e824b83beeeadf63ab4b234/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L633



--
This message was sent by Atlassian Jira
(v8.3.4#803005)