Testing a jvm pipeline on the portability framework locally

[Steve Niemitz <sn...@apache.org>]

If I have a (jvm) pipeline, is there a simple way (ie DirectRunner) to run
it locally but using the portability framework?  I'm running into a lot of
weird bugs running it on dataflow (v2) and want to be able to run it
locally for a faster debug loop.

Re: Testing a jvm pipeline on the portability framework locally

[Alexey Romanenko <ar...@gmail.com>]

Yes, you can do it locally, for example, by running your pipeline via portable Spark or Flink runner. See the details here [1] (select "Portable (Java/Python/Go)” tab for proper code/examples) and here [2].

And the moment of self-advertisement - I gave a talk about cross-language pipeline with a portable runner [3] a while ago, so it may be helpful as well.

[1] https://beam.apache.org/documentation/runners/spark/
[2] https://beam.apache.org/documentation/runners/flink/
[3] https://2020.beamsummit.org/sessions/cross-language-pipeline-python-java/

—
Alexey


> On 18 Jan 2022, at 15:14, Steve Niemitz <sn...@apache.org> wrote:
> 
> If I have a (jvm) pipeline, is there a simple way (ie DirectRunner) to run it locally but using the portability framework?  I'm running into a lot of weird bugs running it on dataflow (v2) and want to be able to run it locally for a faster debug loop.