You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Gyula Fóra <gy...@apache.org> on 2014/09/26 23:21:43 UTC
Basic lambda prototype :)
Hey guys,
I am trying contain my excitement, but I am failing :)
So today I implemented a basic prototype for connecting batch processing
and streaming jobs as a good starting point for a 'lambda architecture
api'. And most surprisingly it works :) It is very simple so far (currently
only works for map operator and chaining is turned off) and some hacking
along the way, but I hope you will like it.
You can see the commit here:
https://github.com/mbalassi/incubator-flink/commit/7bfc16a7803e1f2cc4b38c746af069aab4450637
And a sample job:
https://github.com/mbalassi/incubator-flink/blob/7bfc16a7803e1f2cc4b38c746af069aab4450637/flink-addons/flink-streaming/flink-streaming-core/src/test/java/org/apache/flink/streaming/api/LambdaTest.java
And for the others, it looks something like this:
ExecutionEnvironment batchEnv = ExecutionEnvironment.
*createLocalEnvironment*(1);
StreamExecutionEnvironment streamEnv = StreamExecutionEnvironment.
*createLocalEnvironment*(1);
DataSet<Long> dataSet = batchEnv.generateSequence(1, 10).map(*new*
Increment());
DataStream<Long> dataStream = streamEnv.generateSequence(10, 20);
*dataStream.lambdaJoin(dataSet).map(new Increment()).print();*
dataSet.print();
* LambdaEnvironment lambdaEnv = new LambdaEnvironment(batchEnv,
streamEnv);*
* lambdaEnv.execute(":)");*
I hope I could get you excited too :)
Cheers,
Gyula
Re: Basic lambda prototype :)
Posted by Paris Carbone <pa...@kth.se>.
+1
Looks nice!
On 27 Sep 2014, at 21:04, Kostas Tzoumas <kt...@apache.org>> wrote:
Wow so cool!
On Sat, Sep 27, 2014 at 2:52 PM, Stephan Ewen <se...@apache.org>> wrote:
Wow, amazing :-) And so quickly done!
Am 27.09.2014 11:14 schrieb "Ufuk Celebi" <uc...@apache.org>>:
On 27 Sep 2014, at 08:00, Márton Balassi <mb...@apache.org>> wrote:
I hope I could get you excited too :)
+1 :-)
Re: Basic lambda prototype :)
Posted by Kostas Tzoumas <kt...@apache.org>.
Wow so cool!
On Sat, Sep 27, 2014 at 2:52 PM, Stephan Ewen <se...@apache.org> wrote:
> Wow, amazing :-) And so quickly done!
> Am 27.09.2014 11:14 schrieb "Ufuk Celebi" <uc...@apache.org>:
>
> >
> > On 27 Sep 2014, at 08:00, Márton Balassi <mb...@apache.org> wrote:
> >
> > >> I hope I could get you excited too :)
> >
> > +1 :-)
>
Re: Basic lambda prototype :)
Posted by Stephan Ewen <se...@apache.org>.
Wow, amazing :-) And so quickly done!
Am 27.09.2014 11:14 schrieb "Ufuk Celebi" <uc...@apache.org>:
>
> On 27 Sep 2014, at 08:00, Márton Balassi <mb...@apache.org> wrote:
>
> >> I hope I could get you excited too :)
>
> +1 :-)
Re: Basic lambda prototype :)
Posted by Ufuk Celebi <uc...@apache.org>.
On 27 Sep 2014, at 08:00, Márton Balassi <mb...@apache.org> wrote:
>> I hope I could get you excited too :)
+1 :-)
Re: Basic lambda prototype :)
Posted by Márton Balassi <mb...@apache.org>.
Wow, that is in fact quite impressive. :)
On Fri, Sep 26, 2014 at 11:21 PM, Gyula Fóra <gy...@apache.org> wrote:
> Hey guys,
>
> I am trying contain my excitement, but I am failing :)
>
> So today I implemented a basic prototype for connecting batch processing
> and streaming jobs as a good starting point for a 'lambda architecture
> api'. And most surprisingly it works :) It is very simple so far (currently
> only works for map operator and chaining is turned off) and some hacking
> along the way, but I hope you will like it.
>
> You can see the commit here:
>
> https://github.com/mbalassi/incubator-flink/commit/7bfc16a7803e1f2cc4b38c746af069aab4450637
>
> And a sample job:
>
> https://github.com/mbalassi/incubator-flink/blob/7bfc16a7803e1f2cc4b38c746af069aab4450637/flink-addons/flink-streaming/flink-streaming-core/src/test/java/org/apache/flink/streaming/api/LambdaTest.java
>
> And for the others, it looks something like this:
>
> ExecutionEnvironment batchEnv = ExecutionEnvironment.
> *createLocalEnvironment*(1);
> StreamExecutionEnvironment streamEnv = StreamExecutionEnvironment.
> *createLocalEnvironment*(1);
>
> DataSet<Long> dataSet = batchEnv.generateSequence(1, 10).map(*new*
> Increment());
>
> DataStream<Long> dataStream = streamEnv.generateSequence(10, 20);
>
> *dataStream.lambdaJoin(dataSet).map(new Increment()).print();*
> dataSet.print();
>
> * LambdaEnvironment lambdaEnv = new LambdaEnvironment(batchEnv,
> streamEnv);*
>
> * lambdaEnv.execute(":)");*
>
>
> I hope I could get you excited too :)
>
> Cheers,
>
> Gyula
>