You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Wei Yan <yw...@gmail.com> on 2018/10/15 22:21:39 UTC
Python SDK support for Flink runner
Hi, folks,
Would like a quick check on Python SDK support for Flink runner. I saw
subtasks of BEAM-2889 <https://jira.apache.org/jira/browse/BEAM-2889> have
been resolved. But the getting_started
<https://beam.apache.org/get-started/quickstart-py/> page shows python SDK
only supports local and dataflow. So would like to check progress here, or
any other work left here.
-Wei
Re: Python SDK support for Flink runner
Posted by Wei Yan <yw...@gmail.com>.
Gotcha. Thanks again, Ankur. I'll spend some time trying existing
solutions, and will contribute some code if I can help.
-Wei
On Mon, Oct 15, 2018 at 3:43 PM Ankur Goenka <go...@google.com> wrote:
> Spark runner status is correctly reflected on the Beam website
> https://beam.apache.org/documentation/runners/capability-matrix/
> We don't support python on Spark runner yet.
> However we have added a lot of components which will be used to support
> Python on Spark runner as part of building Flink runner. Let me know if you
> are interested in contributing to expedite spark runner :)
>
> On Mon, Oct 15, 2018 at 3:34 PM Wei Yan <yw...@gmail.com> wrote:
>
>> Great. Thanks, Ankur. I'll give a try.
>> BTW, do you happen to know the progress on spark runner support side?
>>
>> -Wei
>>
>> On Mon, Oct 15, 2018 at 3:29 PM Ankur Goenka <go...@google.com> wrote:
>>
>>> Hi Wei,
>>>
>>> We have made substantial progress towards supporting Python on Flink
>>> runner though its not production ready yet.
>>> We will soon be updating the beam website to better reflect Flink
>>> support but for now here is how you can run python wordcount on Flink
>>> https://beam.apache.org/contribute/portability/#status
>>>
>>> Thanks,
>>> Ankur
>>>
>>> On Mon, Oct 15, 2018 at 3:21 PM Wei Yan <yw...@gmail.com> wrote:
>>>
>>>> Hi, folks,
>>>>
>>>> Would like a quick check on Python SDK support for Flink runner. I saw
>>>> subtasks of BEAM-2889 <https://jira.apache.org/jira/browse/BEAM-2889>
>>>> have been resolved. But the getting_started
>>>> <https://beam.apache.org/get-started/quickstart-py/> page shows python
>>>> SDK only supports local and dataflow. So would like to check progress here,
>>>> or any other work left here.
>>>>
>>>> -Wei
>>>>
>>>
Re: Python SDK support for Flink runner
Posted by Ankur Goenka <go...@google.com>.
Spark runner status is correctly reflected on the Beam website
https://beam.apache.org/documentation/runners/capability-matrix/
We don't support python on Spark runner yet.
However we have added a lot of components which will be used to support
Python on Spark runner as part of building Flink runner. Let me know if you
are interested in contributing to expedite spark runner :)
On Mon, Oct 15, 2018 at 3:34 PM Wei Yan <yw...@gmail.com> wrote:
> Great. Thanks, Ankur. I'll give a try.
> BTW, do you happen to know the progress on spark runner support side?
>
> -Wei
>
> On Mon, Oct 15, 2018 at 3:29 PM Ankur Goenka <go...@google.com> wrote:
>
>> Hi Wei,
>>
>> We have made substantial progress towards supporting Python on Flink
>> runner though its not production ready yet.
>> We will soon be updating the beam website to better reflect Flink support
>> but for now here is how you can run python wordcount on Flink
>> https://beam.apache.org/contribute/portability/#status
>>
>> Thanks,
>> Ankur
>>
>> On Mon, Oct 15, 2018 at 3:21 PM Wei Yan <yw...@gmail.com> wrote:
>>
>>> Hi, folks,
>>>
>>> Would like a quick check on Python SDK support for Flink runner. I saw
>>> subtasks of BEAM-2889 <https://jira.apache.org/jira/browse/BEAM-2889>
>>> have been resolved. But the getting_started
>>> <https://beam.apache.org/get-started/quickstart-py/> page shows python
>>> SDK only supports local and dataflow. So would like to check progress here,
>>> or any other work left here.
>>>
>>> -Wei
>>>
>>
Re: Python SDK support for Flink runner
Posted by Wei Yan <yw...@gmail.com>.
Great. Thanks, Ankur. I'll give a try.
BTW, do you happen to know the progress on spark runner support side?
-Wei
On Mon, Oct 15, 2018 at 3:29 PM Ankur Goenka <go...@google.com> wrote:
> Hi Wei,
>
> We have made substantial progress towards supporting Python on Flink
> runner though its not production ready yet.
> We will soon be updating the beam website to better reflect Flink support
> but for now here is how you can run python wordcount on Flink
> https://beam.apache.org/contribute/portability/#status
>
> Thanks,
> Ankur
>
> On Mon, Oct 15, 2018 at 3:21 PM Wei Yan <yw...@gmail.com> wrote:
>
>> Hi, folks,
>>
>> Would like a quick check on Python SDK support for Flink runner. I saw
>> subtasks of BEAM-2889 <https://jira.apache.org/jira/browse/BEAM-2889>
>> have been resolved. But the getting_started
>> <https://beam.apache.org/get-started/quickstart-py/> page shows python
>> SDK only supports local and dataflow. So would like to check progress here,
>> or any other work left here.
>>
>> -Wei
>>
>
Re: Python SDK support for Flink runner
Posted by Ankur Goenka <go...@google.com>.
Hi Wei,
We have made substantial progress towards supporting Python on Flink runner
though its not production ready yet.
We will soon be updating the beam website to better reflect Flink support
but for now here is how you can run python wordcount on Flink
https://beam.apache.org/contribute/portability/#status
Thanks,
Ankur
On Mon, Oct 15, 2018 at 3:21 PM Wei Yan <yw...@gmail.com> wrote:
> Hi, folks,
>
> Would like a quick check on Python SDK support for Flink runner. I saw
> subtasks of BEAM-2889 <https://jira.apache.org/jira/browse/BEAM-2889>
> have been resolved. But the getting_started
> <https://beam.apache.org/get-started/quickstart-py/> page shows python
> SDK only supports local and dataflow. So would like to check progress here,
> or any other work left here.
>
> -Wei
>