You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Alexander Smirnov <al...@gmail.com> on 2016/04/22 18:41:32 UTC
Flink program without a line of code
Hi guys!
I’m new to Flink, and actually to this mailing list as well :) this is my first message.
I’m still reading the documentation and I would say Flink is an amazing system!! Thanks everybody who participated in the development!
The information I didn’t find in the documentation - if it is possible to describe data(stream) transformation without any code (Java/Scala).
I mean if it is possible to describe datasource functions, all of the operators, connections between them, and sinks in a plain text configuration file and then feed it to Flink.
In this case it would be possible to change data flow without recompilation/redeployment.
Is there a similar functionality in Flink? May be some third party plugin?
Thank you,
Alex
Re: Flink program without a line of code
Posted by Alexander Smirnov <al...@gmail.com>.
thank you so much for the responses, guys!
On Sat, Apr 23, 2016 at 12:09 AM Flavio Pompermaier <po...@okkam.it>
wrote:
> Hi Alexander,
> since I was looking for something similar some days ago here is what I
> know about this argument:
> during the Stratosphere project there was Meteor and Supremo allowing that
> [1] but then it was dismissed in favour of Pig integration that I don't
> wheter it was ever completed. Yiu could give a try to Piglet project[2]
> that allows to use PIG with Spark and Flink but I don't know how well it
> works (Flink integration is also very recent and not documented anywhere).
>
> Best,
> Flavio
>
> [1] http://stratosphere.eu/assets/papers/Sopremo_Meteor%20BigData.pdf
> [2] https://github.com/ksattler/piglet
> On 23 Apr 2016 07:48, "Aljoscha Krettek" <al...@apache.org> wrote:
>
> Hi,
> I think if the Table API/SQL API evolves enough it should be able to
> supply a Flink program as just an SQL query with source/sink definitions.
> Hopefully, in the future. :-)
>
> Cheers,
> Aljoscha
>
> On Fri, 22 Apr 2016 at 23:10 Fabian Hueske <fh...@gmail.com> wrote:
>
>> Hi Alex,
>>
>> welcome to the Flink community!
>> Right now, there is no way to specify a Flink program without writing
>> code (Java, Scala, Python(beta)).
>>
>> In principle it is possible to put such functionality on top of the
>> DataStream or DataSet APIs.
>> This has been done before for other programming APIs (Flink's own
>> libraries Table API, Gelly, FlinkML, and externals Apache Beam / Google
>> DataFlow, Mahout, Cascading, ...). However, all of these are again
>> programming APIs, some specialized for certain use-cases.
>>
>> Specifying Flink programs by config files (or graphically) would require
>> a data model, a DataStream/DataSet program generator and probably a code
>> generation component.
>>
>> Best, Fabian
>>
>> 2016-04-22 18:41 GMT+02:00 Alexander Smirnov <
>> alexander.smirnoff@gmail.com>:
>>
>>> Hi guys!
>>>
>>> I’m new to Flink, and actually to this mailing list as well :) this is
>>> my first message.
>>> I’m still reading the documentation and I would say Flink is an amazing
>>> system!! Thanks everybody who participated in the development!
>>>
>>> The information I didn’t find in the documentation - if it is possible
>>> to describe data(stream) transformation without any code (Java/Scala).
>>> I mean if it is possible to describe datasource functions, all of the
>>> operators, connections between them, and sinks in a plain text
>>> configuration file and then feed it to Flink.
>>> In this case it would be possible to change data flow without
>>> recompilation/redeployment.
>>>
>>> Is there a similar functionality in Flink? May be some third party
>>> plugin?
>>>
>>> Thank you,
>>> Alex
>>
>>
>>
Re: Flink program without a line of code
Posted by Flavio Pompermaier <po...@okkam.it>.
Hi Alexander,
since I was looking for something similar some days ago here is what I know
about this argument:
during the Stratosphere project there was Meteor and Supremo allowing that
[1] but then it was dismissed in favour of Pig integration that I don't
wheter it was ever completed. Yiu could give a try to Piglet project[2]
that allows to use PIG with Spark and Flink but I don't know how well it
works (Flink integration is also very recent and not documented anywhere).
Best,
Flavio
[1] http://stratosphere.eu/assets/papers/Sopremo_Meteor%20BigData.pdf
[2] https://github.com/ksattler/piglet
On 23 Apr 2016 07:48, "Aljoscha Krettek" <al...@apache.org> wrote:
Hi,
I think if the Table API/SQL API evolves enough it should be able to supply
a Flink program as just an SQL query with source/sink definitions.
Hopefully, in the future. :-)
Cheers,
Aljoscha
On Fri, 22 Apr 2016 at 23:10 Fabian Hueske <fh...@gmail.com> wrote:
> Hi Alex,
>
> welcome to the Flink community!
> Right now, there is no way to specify a Flink program without writing code
> (Java, Scala, Python(beta)).
>
> In principle it is possible to put such functionality on top of the
> DataStream or DataSet APIs.
> This has been done before for other programming APIs (Flink's own
> libraries Table API, Gelly, FlinkML, and externals Apache Beam / Google
> DataFlow, Mahout, Cascading, ...). However, all of these are again
> programming APIs, some specialized for certain use-cases.
>
> Specifying Flink programs by config files (or graphically) would require a
> data model, a DataStream/DataSet program generator and probably a code
> generation component.
>
> Best, Fabian
>
> 2016-04-22 18:41 GMT+02:00 Alexander Smirnov <alexander.smirnoff@gmail.com
> >:
>
>> Hi guys!
>>
>> I’m new to Flink, and actually to this mailing list as well :) this is my
>> first message.
>> I’m still reading the documentation and I would say Flink is an amazing
>> system!! Thanks everybody who participated in the development!
>>
>> The information I didn’t find in the documentation - if it is possible to
>> describe data(stream) transformation without any code (Java/Scala).
>> I mean if it is possible to describe datasource functions, all of the
>> operators, connections between them, and sinks in a plain text
>> configuration file and then feed it to Flink.
>> In this case it would be possible to change data flow without
>> recompilation/redeployment.
>>
>> Is there a similar functionality in Flink? May be some third party plugin?
>>
>> Thank you,
>> Alex
>
>
>
Re: Flink program without a line of code
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
I think if the Table API/SQL API evolves enough it should be able to supply
a Flink program as just an SQL query with source/sink definitions.
Hopefully, in the future. :-)
Cheers,
Aljoscha
On Fri, 22 Apr 2016 at 23:10 Fabian Hueske <fh...@gmail.com> wrote:
> Hi Alex,
>
> welcome to the Flink community!
> Right now, there is no way to specify a Flink program without writing code
> (Java, Scala, Python(beta)).
>
> In principle it is possible to put such functionality on top of the
> DataStream or DataSet APIs.
> This has been done before for other programming APIs (Flink's own
> libraries Table API, Gelly, FlinkML, and externals Apache Beam / Google
> DataFlow, Mahout, Cascading, ...). However, all of these are again
> programming APIs, some specialized for certain use-cases.
>
> Specifying Flink programs by config files (or graphically) would require a
> data model, a DataStream/DataSet program generator and probably a code
> generation component.
>
> Best, Fabian
>
> 2016-04-22 18:41 GMT+02:00 Alexander Smirnov <alexander.smirnoff@gmail.com
> >:
>
>> Hi guys!
>>
>> I’m new to Flink, and actually to this mailing list as well :) this is my
>> first message.
>> I’m still reading the documentation and I would say Flink is an amazing
>> system!! Thanks everybody who participated in the development!
>>
>> The information I didn’t find in the documentation - if it is possible to
>> describe data(stream) transformation without any code (Java/Scala).
>> I mean if it is possible to describe datasource functions, all of the
>> operators, connections between them, and sinks in a plain text
>> configuration file and then feed it to Flink.
>> In this case it would be possible to change data flow without
>> recompilation/redeployment.
>>
>> Is there a similar functionality in Flink? May be some third party plugin?
>>
>> Thank you,
>> Alex
>
>
>
Re: Flink program without a line of code
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Alex,
welcome to the Flink community!
Right now, there is no way to specify a Flink program without writing code
(Java, Scala, Python(beta)).
In principle it is possible to put such functionality on top of the
DataStream or DataSet APIs.
This has been done before for other programming APIs (Flink's own libraries
Table API, Gelly, FlinkML, and externals Apache Beam / Google DataFlow,
Mahout, Cascading, ...). However, all of these are again programming APIs,
some specialized for certain use-cases.
Specifying Flink programs by config files (or graphically) would require a
data model, a DataStream/DataSet program generator and probably a code
generation component.
Best, Fabian
2016-04-22 18:41 GMT+02:00 Alexander Smirnov <al...@gmail.com>:
> Hi guys!
>
> I’m new to Flink, and actually to this mailing list as well :) this is my
> first message.
> I’m still reading the documentation and I would say Flink is an amazing
> system!! Thanks everybody who participated in the development!
>
> The information I didn’t find in the documentation - if it is possible to
> describe data(stream) transformation without any code (Java/Scala).
> I mean if it is possible to describe datasource functions, all of the
> operators, connections between them, and sinks in a plain text
> configuration file and then feed it to Flink.
> In this case it would be possible to change data flow without
> recompilation/redeployment.
>
> Is there a similar functionality in Flink? May be some third party plugin?
>
> Thank you,
> Alex