You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by amir bahmanyari <am...@yahoo.com.INVALID> on 2017/04/04 00:11:44 UTC
Flink limitations under Beam
Hi colleagues, been a long time. New project.What feature(s)/capabilities of Flink would become unavailable/limited if the pipeline app is written in Beam sdk using FlinkRunner?
Thanks+regards
Re: Flink limitations under Beam
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
I don’t have any hard numbers for this, and it depends on what kind of state backend is used (Heap backend or RocksDB backend).
Iterations are documented in the Flink doc: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/datastream_api.html#iterations <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/datastream_api.html#iterations> Essentially what they allow is to have a processing graph that is not a DAG. This can be useful in machine learning use cases where you want to feed back some information to an earlier operation to update a model.
Best,
Aljoscha
> On 4. Apr 2017, at 09:46, amir bahmanyari <am...@yahoo.com> wrote:
>
> Thanks Aljoscha,
> How sever of performance difference are we talking about here?
> examples of "Iterations/feedback edges" and how they get applied?
> Thanks so much Aljoscha.
>
>
>
> From: Aljoscha Krettek <al...@apache.org>
> To: dev@flink.apache.org; amir bahmanyari <am...@yahoo.com>
> Sent: Tuesday, April 4, 2017 1:59 AM
> Subject: Re: Flink limitations under Beam
>
> Hi,
> I think you would mainly lose on performance, since Beam adds an additional layer of abstraction and has some requirements from Runners that slow things down compared to running natively on Flink.
>
> I think feature wise, the only thing that you will lose is support for Iterations/feedback edges. Flink has this while Beam does not (yet) have an API for that.
>
> Best,
> Aljoscha
>
> > On 4. Apr 2017, at 02:11, amir bahmanyari <amirtousa@yahoo.com.INVALID <ma...@yahoo.com.INVALID>> wrote:
> >
> > Hi colleagues, been a long time. New project.What feature(s)/capabilities of Flink would become unavailable/limited if the pipeline app is written in Beam sdk using FlinkRunner?
> > Thanks+regards
>
>
Re: Flink limitations under Beam
Posted by amir bahmanyari <am...@yahoo.com.INVALID>.
Thanks Aljoscha,How sever of performance difference are we talking about here? examples of "Iterations/feedback edges" and how they get applied?Thanks so much Aljoscha.
From: Aljoscha Krettek <al...@apache.org>
To: dev@flink.apache.org; amir bahmanyari <am...@yahoo.com>
Sent: Tuesday, April 4, 2017 1:59 AM
Subject: Re: Flink limitations under Beam
Hi,
I think you would mainly lose on performance, since Beam adds an additional layer of abstraction and has some requirements from Runners that slow things down compared to running natively on Flink.
I think feature wise, the only thing that you will lose is support for Iterations/feedback edges. Flink has this while Beam does not (yet) have an API for that.
Best,
Aljoscha
> On 4. Apr 2017, at 02:11, amir bahmanyari <am...@yahoo.com.INVALID> wrote:
>
> Hi colleagues, been a long time. New project.What feature(s)/capabilities of Flink would become unavailable/limited if the pipeline app is written in Beam sdk using FlinkRunner?
> Thanks+regards
Re: Flink limitations under Beam
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
I think you would mainly lose on performance, since Beam adds an additional layer of abstraction and has some requirements from Runners that slow things down compared to running natively on Flink.
I think feature wise, the only thing that you will lose is support for Iterations/feedback edges. Flink has this while Beam does not (yet) have an API for that.
Best,
Aljoscha
> On 4. Apr 2017, at 02:11, amir bahmanyari <am...@yahoo.com.INVALID> wrote:
>
> Hi colleagues, been a long time. New project.What feature(s)/capabilities of Flink would become unavailable/limited if the pipeline app is written in Beam sdk using FlinkRunner?
> Thanks+regards