You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Ashutosh Kumar <km...@gmail.com> on 2016/05/26 07:20:01 UTC
Apache Beam and Flink
How does apache beam fits with flink ? Is it an alternative for flink or
complementary to it ?
Thanks
Ashutosh
Re: Apache Beam and Flink
Posted by Maximilian Michels <mx...@apache.org>.
Small addition: The Flink Runner translates into the DataSet or
DataStream API depending on the "streaming" flag of the
PipelineOptions. The default mode is batch. Ultimately, this flag we
be removed and replaced with an automated decision depending on the
sources used.
On Thu, May 26, 2016 at 10:20 AM, Slim Baltagi <sb...@gmail.com> wrote:
> Hi Ashutosh
>
> There is a related open JIRA: Enable DataSet and DataStream Joins
> https://issues.apache.org/jira/browse/FLINK-2320
>
> Slim
>
>
> On May 26, 2016, at 3:05 AM, Fabian Hueske <fh...@gmail.com> wrote:
>
> No, that is not supported yet.
> Beam provides a common API but the Flink runner translates programs against
> batch sources into the DataSet API programs and Beam programs against
> streaming source into DataStream programs.
> It is not possible to mix both.
>
> 2016-05-26 10:00 GMT+02:00 Ashutosh Kumar <km...@gmail.com>:
>>
>> Thanks . So if we use Beam API with flink engine then we can get inter
>> action between batch and stream ? As i know currently in flink Dataset and
>> DStream can not talk . Is this correct ?
>> Thanks
>> Ashutosh
>>
>>
>> On Thu, May 26, 2016 at 1:09 PM, Slim Baltagi <sb...@gmail.com> wrote:
>>>
>>> Hi Ashutosh
>>>
>>> Apache Beam provides a Unified API for batch and streaming.
>>> It also supports multiple ‘runners’: local, Apache Spark, Apache Flink
>>> and Google Cloud Data Flow (commercial service).
>>> It is not an alternative to Flink because it is an API and you still need
>>> an execution engine.
>>> It can be used as an alternative API to using the two Flink APIs :
>>> DataSet API and DataStream API.
>>> It can be complementary to Flink in the way that you use Beam as API and
>>> Flink as the execution engine.
>>> Many of Flink committers are also Apache Beam committers!
>>> The following blogs describe why Apache Beam:
>>>
>>> from Flink perspective: http://data-artisans.com/why-apache-beam/
>>> from Google perspective.
>>> https://cloud.google.com/blog/big-data/2016/05/why-apache-beam-a-google-perspective
>>>
>>>
>>> A few recent resources about Apache Beam published this month: May 2016
>>>
>>> Running Apache Beam (screencast)
>>> https://www.youtube.com/watch?v=dwxUbzbwtyI
>>> Introduction to Apache Beam ( presentation)
>>> https://skillsmatter.com/skillscasts/8036-apache-flink-may-meetup
>>> Introduction to Apache Beam ( blog)
>>> http://www.talend.com/blog/2016/05/02/introduction-to-apache-beam
>>>
>>>
>>> I hope this helps.
>>>
>>> Thanks
>>>
>>> Slim Baltagi
>>>
>>> On May 26, 2016, at 2:20 AM, Ashutosh Kumar <km...@gmail.com>
>>> wrote:
>>>
>>> How does apache beam fits with flink ? Is it an alternative for flink or
>>> complementary to it ?
>>>
>>> Thanks
>>> Ashutosh
>>>
>>>
>>
>
>
Re: Apache Beam and Flink
Posted by Slim Baltagi <sb...@gmail.com>.
Hi Ashutosh
There is a related open JIRA: Enable DataSet and DataStream Joins https://issues.apache.org/jira/browse/FLINK-2320 <https://issues.apache.org/jira/browse/FLINK-2320>
Slim
> On May 26, 2016, at 3:05 AM, Fabian Hueske <fh...@gmail.com> wrote:
>
> No, that is not supported yet.
> Beam provides a common API but the Flink runner translates programs against batch sources into the DataSet API programs and Beam programs against streaming source into DataStream programs.
> It is not possible to mix both.
>
> 2016-05-26 10:00 GMT+02:00 Ashutosh Kumar <kmr.ashutosh16@gmail.com <ma...@gmail.com>>:
> Thanks . So if we use Beam API with flink engine then we can get inter action between batch and stream ? As i know currently in flink Dataset and DStream can not talk . Is this correct ?
> Thanks
> Ashutosh
>
>
> On Thu, May 26, 2016 at 1:09 PM, Slim Baltagi <sbaltagi@gmail.com <ma...@gmail.com>> wrote:
> Hi Ashutosh
>
> Apache Beam provides a Unified API for batch and streaming.
> It also supports multiple ‘runners’: local, Apache Spark, Apache Flink and Google Cloud Data Flow (commercial service).
> It is not an alternative to Flink because it is an API and you still need an execution engine.
> It can be used as an alternative API to using the two Flink APIs : DataSet API and DataStream API.
> It can be complementary to Flink in the way that you use Beam as API and Flink as the execution engine.
> Many of Flink committers are also Apache Beam committers!
> The following blogs describe why Apache Beam:
> from Flink perspective: http://data-artisans.com/why-apache-beam/ <http://data-artisans.com/why-apache-beam/>
> from Google perspective. https://cloud.google.com/blog/big-data/2016/05/why-apache-beam-a-google-perspective <https://cloud.google.com/blog/big-data/2016/05/why-apache-beam-a-google-perspective>
>
> A few recent resources about Apache Beam published this month: May 2016
> Running Apache Beam (screencast) https://www.youtube.com/watch?v=dwxUbzbwtyI <https://www.youtube.com/watch?v=dwxUbzbwtyI>
> Introduction to Apache Beam ( presentation) https://skillsmatter.com/skillscasts/8036-apache-flink-may-meetup <https://skillsmatter.com/skillscasts/8036-apache-flink-may-meetup>
> Introduction to Apache Beam ( blog) http://www.talend.com/blog/2016/05/02/introduction-to-apache-beam <http://www.talend.com/blog/2016/05/02/introduction-to-apache-beam>
>
> I hope this helps.
>
> Thanks
>
> Slim Baltagi
>
>> On May 26, 2016, at 2:20 AM, Ashutosh Kumar <kmr.ashutosh16@gmail.com <ma...@gmail.com>> wrote:
>>
>> How does apache beam fits with flink ? Is it an alternative for flink or complementary to it ?
>>
>> Thanks
>> Ashutosh
>
>
>
Re: Apache Beam and Flink
Posted by Fabian Hueske <fh...@gmail.com>.
No, that is not supported yet.
Beam provides a common API but the Flink runner translates programs against
batch sources into the DataSet API programs and Beam programs against
streaming source into DataStream programs.
It is not possible to mix both.
2016-05-26 10:00 GMT+02:00 Ashutosh Kumar <km...@gmail.com>:
> Thanks . So if we use Beam API with flink engine then we can get inter
> action between batch and stream ? As i know currently in flink Dataset and
> DStream can not talk . Is this correct ?
> Thanks
> Ashutosh
>
>
> On Thu, May 26, 2016 at 1:09 PM, Slim Baltagi <sb...@gmail.com> wrote:
>
>> Hi Ashutosh
>>
>> Apache Beam provides a Unified API for batch and streaming.
>> It also supports multiple ‘runners’: local, Apache Spark, Apache Flink
>> and Google Cloud Data Flow (commercial service).
>> It is not an alternative to Flink because it is an API and you still need
>> an execution engine.
>> It can be used as an alternative API to using the two Flink APIs :
>> DataSet API and DataStream API.
>> It can be complementary to Flink in the way that you use Beam as API and
>> Flink as the execution engine.
>> Many of Flink committers are also Apache Beam committers!
>> The following blogs describe why Apache Beam:
>>
>> - from Flink perspective: http://data-artisans.com/why-apache-beam/
>> - from Google perspective.
>> https://cloud.google.com/blog/big-data/2016/05/why-apache-beam-a-google-perspective
>>
>>
>> A few recent resources about Apache Beam published this month: May 2016
>>
>> - Running Apache Beam (screencast)
>> https://www.youtube.com/watch?v=dwxUbzbwtyI
>> - Introduction to Apache Beam ( presentation)
>> https://skillsmatter.com/skillscasts/8036-apache-flink-may-meetup
>> - Introduction to Apache Beam ( blog)
>> http://www.talend.com/blog/2016/05/02/introduction-to-apache-beam
>>
>>
>> I hope this helps.
>>
>> Thanks
>>
>> Slim Baltagi
>>
>> On May 26, 2016, at 2:20 AM, Ashutosh Kumar <km...@gmail.com>
>> wrote:
>>
>> How does apache beam fits with flink ? Is it an alternative for flink or
>> complementary to it ?
>>
>> Thanks
>> Ashutosh
>>
>>
>>
>
Re: Apache Beam and Flink
Posted by Ashutosh Kumar <km...@gmail.com>.
Thanks . So if we use Beam API with flink engine then we can get inter
action between batch and stream ? As i know currently in flink Dataset and
DStream can not talk . Is this correct ?
Thanks
Ashutosh
On Thu, May 26, 2016 at 1:09 PM, Slim Baltagi <sb...@gmail.com> wrote:
> Hi Ashutosh
>
> Apache Beam provides a Unified API for batch and streaming.
> It also supports multiple ‘runners’: local, Apache Spark, Apache Flink and
> Google Cloud Data Flow (commercial service).
> It is not an alternative to Flink because it is an API and you still need
> an execution engine.
> It can be used as an alternative API to using the two Flink APIs : DataSet
> API and DataStream API.
> It can be complementary to Flink in the way that you use Beam as API and
> Flink as the execution engine.
> Many of Flink committers are also Apache Beam committers!
> The following blogs describe why Apache Beam:
>
> - from Flink perspective: http://data-artisans.com/why-apache-beam/
> - from Google perspective.
> https://cloud.google.com/blog/big-data/2016/05/why-apache-beam-a-google-perspective
>
>
> A few recent resources about Apache Beam published this month: May 2016
>
> - Running Apache Beam (screencast)
> https://www.youtube.com/watch?v=dwxUbzbwtyI
> - Introduction to Apache Beam ( presentation)
> https://skillsmatter.com/skillscasts/8036-apache-flink-may-meetup
> - Introduction to Apache Beam ( blog)
> http://www.talend.com/blog/2016/05/02/introduction-to-apache-beam
>
>
> I hope this helps.
>
> Thanks
>
> Slim Baltagi
>
> On May 26, 2016, at 2:20 AM, Ashutosh Kumar <km...@gmail.com>
> wrote:
>
> How does apache beam fits with flink ? Is it an alternative for flink or
> complementary to it ?
>
> Thanks
> Ashutosh
>
>
>
Re: Apache Beam and Flink
Posted by Slim Baltagi <sb...@gmail.com>.
Hi Ashutosh
Apache Beam provides a Unified API for batch and streaming.
It also supports multiple ‘runners’: local, Apache Spark, Apache Flink and Google Cloud Data Flow (commercial service).
It is not an alternative to Flink because it is an API and you still need an execution engine.
It can be used as an alternative API to using the two Flink APIs : DataSet API and DataStream API.
It can be complementary to Flink in the way that you use Beam as API and Flink as the execution engine.
Many of Flink committers are also Apache Beam committers!
The following blogs describe why Apache Beam:
from Flink perspective: http://data-artisans.com/why-apache-beam/
from Google perspective. https://cloud.google.com/blog/big-data/2016/05/why-apache-beam-a-google-perspective
A few recent resources about Apache Beam published this month: May 2016
Running Apache Beam (screencast) https://www.youtube.com/watch?v=dwxUbzbwtyI
Introduction to Apache Beam ( presentation) https://skillsmatter.com/skillscasts/8036-apache-flink-may-meetup
Introduction to Apache Beam ( blog) http://www.talend.com/blog/2016/05/02/introduction-to-apache-beam
I hope this helps.
Thanks
Slim Baltagi
> On May 26, 2016, at 2:20 AM, Ashutosh Kumar <km...@gmail.com> wrote:
>
> How does apache beam fits with flink ? Is it an alternative for flink or complementary to it ?
>
> Thanks
> Ashutosh