You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Dante Van den Broeke <Da...@UGent.be> on 2019/07/19 14:40:40 UTC
Jython support for Flink
Dear,
I'm a student currently working on a project involving apache kafka and flink. The project itself is revolved around path prediction and machine learning for websites. To test a prove of concept I setup a kafka server locally (goal is to expend this to a google cloud server or similar later) and a kafka producer (written in java intelliJ idea project). The producer would send JSON data (currently just a local file but later json data from the website itself) to a flink-kafka connection and the data transformation (key-windowed by user id) would than happen in the flink framework.
The problem i'm facing however is that i wrote all the algorithms for transformation of the data in python and i'm struggling with initializing a jython environment to setup the flink-kafka connection.
I was wondering whether or not there is a working example for this setup / some documentation regarding the framework as i'm struggling to find a lot of documentation for my application online.
thanks in advance.
kind regards,
Dante Van den Broeke
Re: Jython support for Flink
Posted by Zili Chen <wa...@gmail.com>.
Hi Dante,
Nice finding! I just miss this powerful project :-)
Best,
tison.
Dante Van den Broeke <Da...@ugent.be> 于2019年7月20日周六 下午5:28写道：
>
> Hi Tison, Jeff,
>
> Thanks a lot for the help! I’ll definitly look into the python API and
> py4j support next. I was also thinking about trying to create the pipeline
> through beam and flink instead of kafka and flink, as i see that python is
> a full class citizen in the beam framework!
>
> Regards,
> Dante
>
> Op 20 jul. 2019 om 01:59 heeft Zili Chen <wa...@gmail.com> het
> volgende geschreven:
>
> Hi Dante,
>
> Both Jython and Jython support for Flink are out of development
> and maintain. As pointed out by Jeff, Flink 1.9 supports Python
> api via py4j[1] and the document page as posted.
>
> I guess your algorithms are written in CPython instead of Jython
> and want Jython only for interoperate, and thus recommend you to
> have a look at the doc posted above.
>
> For previous Jython support example or setup, cc Chesnay who is
> our committer familiar with this scope.
>
> Best,
> tison.
>
> [1] https://www.py4j.org/
>
>
>
> Jeff Zhang <zj...@gmail.com> 于2019年7月19日周五 下午11:06写道：
>
>> Hi Dante,
>>
>> Flink 1.9 support python api, which may be what you want. See
>> https://ci.apache.org/projects/flink/flink-docs-master/tutorials/python_table_api.html
>>
>>
>> Dante Van den Broeke <Da...@ugent.be> 于2019年7月19日周五
>> 下午10:40写道：
>>
>>> Dear,
>>>
>>>
>>> I'm a student currently working on a project involving apache kafka and
>>> flink. The project itself is revolved around path prediction and machine
>>> learning for websites. To test a prove of concept I setup a kafka server
>>> locally (goal is to expend this to a google cloud server or similar later)
>>> and a kafka producer (written in java intelliJ idea project). The producer
>>> would send JSON data (currently just a local file but later json data from
>>> the website itself) to a flink-kafka connection and the data transformation
>>> (key-windowed by user id) would than happen in the flink framework.
>>>
>>>
>>> The problem i'm facing however is that i wrote all the algorithms for
>>> transformation of the data in python and i'm struggling with initializing a
>>> jython environment to setup the flink-kafka connection.
>>>
>>> I was wondering whether or not there is a working example for this setup
>>> / some documentation regarding the framework as i'm struggling to find a
>>> lot of documentation for my application online.
>>>
>>>
>>> thanks in advance.
>>>
>>>
>>> kind regards,
>>>
>>> Dante Van den Broeke
>>>
>>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
Re: Jython support for Flink
Posted by Dante Van den Broeke <Da...@UGent.be>.
Hi Tison, Jeff,
Thanks a lot for the help! I’ll definitly look into the python API and py4j support next. I was also thinking about trying to create the pipeline through beam and flink instead of kafka and flink, as i see that python is a full class citizen in the beam framework!
Regards,
Dante
Op 20 jul. 2019 om 01:59 heeft Zili Chen <wa...@gmail.com>> het volgende geschreven:
Hi Dante,
Both Jython and Jython support for Flink are out of development
and maintain. As pointed out by Jeff, Flink 1.9 supports Python
api via py4j[1] and the document page as posted.
I guess your algorithms are written in CPython instead of Jython
and want Jython only for interoperate, and thus recommend you to
have a look at the doc posted above.
For previous Jython support example or setup, cc Chesnay who is
our committer familiar with this scope.
Best,
tison.
[1] https://www.py4j.org/
Jeff Zhang <zj...@gmail.com>> 于2019年7月19日周五 下午11:06写道：
Hi Dante,
Flink 1.9 support python api, which may be what you want. See https://ci.apache.org/projects/flink/flink-docs-master/tutorials/python_table_api.html
Dante Van den Broeke <Da...@ugent.be>> 于2019年7月19日周五 下午10:40写道：
Dear,
I'm a student currently working on a project involving apache kafka and flink. The project itself is revolved around path prediction and machine learning for websites. To test a prove of concept I setup a kafka server locally (goal is to expend this to a google cloud server or similar later) and a kafka producer (written in java intelliJ idea project). The producer would send JSON data (currently just a local file but later json data from the website itself) to a flink-kafka connection and the data transformation (key-windowed by user id) would than happen in the flink framework.
The problem i'm facing however is that i wrote all the algorithms for transformation of the data in python and i'm struggling with initializing a jython environment to setup the flink-kafka connection.
I was wondering whether or not there is a working example for this setup / some documentation regarding the framework as i'm struggling to find a lot of documentation for my application online.
thanks in advance.
kind regards,
Dante Van den Broeke
--
Best Regards
Jeff Zhang
Re: Jython support for Flink
Posted by Zili Chen <wa...@gmail.com>.
Hi Dante,
Both Jython and Jython support for Flink are out of development
and maintain. As pointed out by Jeff, Flink 1.9 supports Python
api via py4j[1] and the document page as posted.
I guess your algorithms are written in CPython instead of Jython
and want Jython only for interoperate, and thus recommend you to
have a look at the doc posted above.
For previous Jython support example or setup, cc Chesnay who is
our committer familiar with this scope.
Best,
tison.
[1] https://www.py4j.org/
Jeff Zhang <zj...@gmail.com> 于2019年7月19日周五 下午11:06写道：
> Hi Dante,
>
> Flink 1.9 support python api, which may be what you want. See
> https://ci.apache.org/projects/flink/flink-docs-master/tutorials/python_table_api.html
>
>
> Dante Van den Broeke <Da...@ugent.be> 于2019年7月19日周五
> 下午10:40写道：
>
>> Dear,
>>
>>
>> I'm a student currently working on a project involving apache kafka and
>> flink. The project itself is revolved around path prediction and machine
>> learning for websites. To test a prove of concept I setup a kafka server
>> locally (goal is to expend this to a google cloud server or similar later)
>> and a kafka producer (written in java intelliJ idea project). The producer
>> would send JSON data (currently just a local file but later json data from
>> the website itself) to a flink-kafka connection and the data transformation
>> (key-windowed by user id) would than happen in the flink framework.
>>
>>
>> The problem i'm facing however is that i wrote all the algorithms for
>> transformation of the data in python and i'm struggling with initializing a
>> jython environment to setup the flink-kafka connection.
>>
>> I was wondering whether or not there is a working example for this setup
>> / some documentation regarding the framework as i'm struggling to find a
>> lot of documentation for my application online.
>>
>>
>> thanks in advance.
>>
>>
>> kind regards,
>>
>> Dante Van den Broeke
>>
>>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: Jython support for Flink
Posted by Jeff Zhang <zj...@gmail.com>.
Hi Dante,
Flink 1.9 support python api, which may be what you want. See
https://ci.apache.org/projects/flink/flink-docs-master/tutorials/python_table_api.html
Dante Van den Broeke <Da...@ugent.be> 于2019年7月19日周五 下午10:40写道：
> Dear,
>
>
> I'm a student currently working on a project involving apache kafka and
> flink. The project itself is revolved around path prediction and machine
> learning for websites. To test a prove of concept I setup a kafka server
> locally (goal is to expend this to a google cloud server or similar later)
> and a kafka producer (written in java intelliJ idea project). The producer
> would send JSON data (currently just a local file but later json data from
> the website itself) to a flink-kafka connection and the data transformation
> (key-windowed by user id) would than happen in the flink framework.
>
>
> The problem i'm facing however is that i wrote all the algorithms for
> transformation of the data in python and i'm struggling with initializing a
> jython environment to setup the flink-kafka connection.
>
> I was wondering whether or not there is a working example for this setup /
> some documentation regarding the framework as i'm struggling to find a lot
> of documentation for my application online.
>
>
> thanks in advance.
>
>
> kind regards,
>
> Dante Van den Broeke
>
>
--
Best Regards
Jeff Zhang