You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Bing Lin <bi...@gmail.com> on 2018/10/02 17:48:57 UTC

Flink Python streaming

Hi,

I'm wondering how I can add dependencies for third party and custom
libraries to be executed in Flink for python streaming?

Thank you,

Bing

Re: Flink Python streaming

Posted by Chesnay Schepler <ch...@apache.org>.

Hello,

to use libraries you have to supply them when submitting the job as 
described below. Additional directories/files will be placed in the same 
directory as your script on each TM.

See 
https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/python.html#executing-plans

Note that libraries that make use of C-extensions will probably not work.

On 03.10.2018 09:14, Fabian Hueske wrote:
> Hi,
>
> AFAIK it's not that easy. Flink's Python support is based on Jython 
> which translates Python code into JVM byte code. Therefore, native 
> libs are not supported.
>
> Chesnay (in CC) knows the details here.
>
> Best, Fabian
>
> Hequn Cheng <chenghequn@gmail.com <ma...@gmail.com>> 
> schrieb am Mi., 3. Okt. 2018, 04:30:
>
>     Hi Bing,
>
>     I'm not familiar with python programming. I guess we can simply
>     import libraries in the python script. A example can be found here[1].
>     Hope this helps.
>
>     Best, Hequn
>     [1]
>     https://github.com/wdm0006/flink-python-examples/blob/master/mandelbrot/mandelbrot_set.py
>
>     On Wed, Oct 3, 2018 at 1:49 AM Bing Lin <binglin3@gmail.com
>     <ma...@gmail.com>> wrote:
>
>         Hi,
>
>         I'm wondering how I can add dependencies for third party and
>         custom libraries to be executed in Flink for python streaming?
>
>         Thank you,
>
>         Bing
>

Re: Flink Python streaming

Posted by Fabian Hueske <fh...@gmail.com>.

Hi,

AFAIK it's not that easy. Flink's Python support is based on Jython which
translates Python code into JVM byte code. Therefore, native libs are not
supported.

Chesnay (in CC) knows the details here.

Best, Fabian

Hequn Cheng <ch...@gmail.com> schrieb am Mi., 3. Okt. 2018, 04:30:

> Hi Bing,
>
> I'm not familiar with python programming. I guess we can simply import
> libraries in the python script. A example can be found here[1].
> Hope this helps.
>
> Best, Hequn
> [1]
> https://github.com/wdm0006/flink-python-examples/blob/master/mandelbrot/mandelbrot_set.py
>
> On Wed, Oct 3, 2018 at 1:49 AM Bing Lin <bi...@gmail.com> wrote:
>
>> Hi,
>>
>> I'm wondering how I can add dependencies for third party and custom
>> libraries to be executed in Flink for python streaming?
>>
>> Thank you,
>>
>> Bing
>>
>

Re: Flink Python streaming

Posted by Hequn Cheng <ch...@gmail.com>.

Hi Bing,

I'm not familiar with python programming. I guess we can simply import
libraries in the python script. A example can be found here[1].
Hope this helps.

Best, Hequn
[1]
https://github.com/wdm0006/flink-python-examples/blob/master/mandelbrot/mandelbrot_set.py

On Wed, Oct 3, 2018 at 1:49 AM Bing Lin <bi...@gmail.com> wrote:

> Hi,
>
> I'm wondering how I can add dependencies for third party and custom
> libraries to be executed in Flink for python streaming?
>
> Thank you,
>
> Bing
>