You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Nir Gazit <ni...@gmail.com> on 2021/01/14 12:12:02 UTC
Difference between Flink runner and portable runner on Python
Hey,
The documentation lists 2 options for running Python Beam on Flink:
"Portable (Python)" and "Portable (Python / Go / Java)". What are the
differences between them? which one is recommended?
Thanks!
Nir
Re: Difference between Flink runner and portable runner on Python
Posted by Ankur Goenka <go...@google.com>.
DOCKER is the recommended way but for testing and development you can also
use LOOPBACK.
More details here
https://github.com/apache/beam/blob/17fbe8be70b71bc1236e553a5fe7902a2df5dda4/sdks/python/apache_beam/options/pipeline_options.py#L1115
On Thu, Jan 14, 2021 at 11:40 AM Nir Gazit <ni...@gmail.com> wrote:
> Thanks! And is there a recommended approach for deploying the SDK harness?
>
> On Thu, Jan 14, 2021 at 7:19 PM Kyle Weaver <kc...@google.com> wrote:
>
>> It's mostly just a different entry point. With (Python / Go / Java), the
>> user has to start a job server themselves before submitting a job. With
>> (Python), the job server is managed automatically in the background. Aside
>> from that, though, there's very little difference. We recommend (Python) to
>> new users only because it's a smidge easier to set up.
>>
>> On Thu, Jan 14, 2021 at 4:12 AM Nir Gazit <ni...@gmail.com> wrote:
>>
>>> Hey,
>>> The documentation lists 2 options for running Python Beam on Flink:
>>> "Portable (Python)" and "Portable (Python / Go / Java)". What are the
>>> differences between them? which one is recommended?
>>>
>>> Thanks!
>>> Nir
>>>
>>
Re: Difference between Flink runner and portable runner on Python
Posted by Nir Gazit <ni...@gmail.com>.
Thanks! And is there a recommended approach for deploying the SDK harness?
On Thu, Jan 14, 2021 at 7:19 PM Kyle Weaver <kc...@google.com> wrote:
> It's mostly just a different entry point. With (Python / Go / Java), the
> user has to start a job server themselves before submitting a job. With
> (Python), the job server is managed automatically in the background. Aside
> from that, though, there's very little difference. We recommend (Python) to
> new users only because it's a smidge easier to set up.
>
> On Thu, Jan 14, 2021 at 4:12 AM Nir Gazit <ni...@gmail.com> wrote:
>
>> Hey,
>> The documentation lists 2 options for running Python Beam on Flink:
>> "Portable (Python)" and "Portable (Python / Go / Java)". What are the
>> differences between them? which one is recommended?
>>
>> Thanks!
>> Nir
>>
>
Re: Difference between Flink runner and portable runner on Python
Posted by Kyle Weaver <kc...@google.com>.
It's mostly just a different entry point. With (Python / Go / Java), the
user has to start a job server themselves before submitting a job. With
(Python), the job server is managed automatically in the background. Aside
from that, though, there's very little difference. We recommend (Python) to
new users only because it's a smidge easier to set up.
On Thu, Jan 14, 2021 at 4:12 AM Nir Gazit <ni...@gmail.com> wrote:
> Hey,
> The documentation lists 2 options for running Python Beam on Flink:
> "Portable (Python)" and "Portable (Python / Go / Java)". What are the
> differences between them? which one is recommended?
>
> Thanks!
> Nir
>