You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Jin Yi <el...@gmail.com> on 2020/02/27 05:46:17 UTC
Apache Beam Side input vs Flink Broadcast Stream
Hi All,
there is a recent published article in the flink official website for
running beam on top of flink
https://flink.apache.org/ecosystem/2020/02/22/apache-beam-how-beam-runs-on-top-of-flink.html
In the article:
- You get additional features like side inputs and cross-language
pipelines that are not supported natively in Flink but only supported when
using Beam with Flink
Ultimately, Beam pipeline will be translated into Flink job. So does beam's
side input translates into Flink Broadcast stream?
If I look at org.apache.beam.runners.flink.FlinkStreamingTransformTranslators,
it looks like converting the the side input into broadcast stream, then why
it says Flink does not support it natively?
Thanks a lot!
Eleanore
Re: Apache Beam Side input vs Flink Broadcast Stream
Posted by Jin Yi <el...@gmail.com>.
Hi Arvid,
Thanks a lot for the response and yes I am aware of FLIP-17.
Eleanore
On Fri, Feb 28, 2020 at 2:16 AM Arvid Heise <ar...@ververica.com> wrote:
> Hi Eleanore,
>
> we understand side-input as something more general than simple broadcast
> input, see FLIP-17 for details [1].
>
> If a broadcast fits your use case, you can use that of course. We are
> aiming for something, where a side input can also be co-partitioned. We are
> currently laying the foundations for that feature.
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API
>
> On Thu, Feb 27, 2020 at 6:46 AM Jin Yi <el...@gmail.com> wrote:
>
>> Hi All,
>>
>> there is a recent published article in the flink official website for
>> running beam on top of flink
>> https://flink.apache.org/ecosystem/2020/02/22/apache-beam-how-beam-runs-on-top-of-flink.html
>>
>> In the article:
>>
>> - You get additional features like side inputs and cross-language
>> pipelines that are not supported natively in Flink but only supported when
>> using Beam with Flink
>>
>> Ultimately, Beam pipeline will be translated into Flink job. So does
>> beam's side input translates into Flink Broadcast stream?
>>
>> If I look at org.apache.beam.runners.flink.FlinkStreamingTransformTranslators,
>> it looks like converting the the side input into broadcast stream, then why
>> it says Flink does not support it natively?
>>
>> Thanks a lot!
>> Eleanore
>>
>>
Re: Apache Beam Side input vs Flink Broadcast Stream
Posted by Jin Yi <el...@gmail.com>.
Hi Arvid,
Thanks a lot for the response and yes I am aware of FLIP-17.
Eleanore
On Fri, Feb 28, 2020 at 2:16 AM Arvid Heise <ar...@ververica.com> wrote:
> Hi Eleanore,
>
> we understand side-input as something more general than simple broadcast
> input, see FLIP-17 for details [1].
>
> If a broadcast fits your use case, you can use that of course. We are
> aiming for something, where a side input can also be co-partitioned. We are
> currently laying the foundations for that feature.
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API
>
> On Thu, Feb 27, 2020 at 6:46 AM Jin Yi <el...@gmail.com> wrote:
>
>> Hi All,
>>
>> there is a recent published article in the flink official website for
>> running beam on top of flink
>> https://flink.apache.org/ecosystem/2020/02/22/apache-beam-how-beam-runs-on-top-of-flink.html
>>
>> In the article:
>>
>> - You get additional features like side inputs and cross-language
>> pipelines that are not supported natively in Flink but only supported when
>> using Beam with Flink
>>
>> Ultimately, Beam pipeline will be translated into Flink job. So does
>> beam's side input translates into Flink Broadcast stream?
>>
>> If I look at org.apache.beam.runners.flink.FlinkStreamingTransformTranslators,
>> it looks like converting the the side input into broadcast stream, then why
>> it says Flink does not support it natively?
>>
>> Thanks a lot!
>> Eleanore
>>
>>
Re: Apache Beam Side input vs Flink Broadcast Stream
Posted by Arvid Heise <ar...@ververica.com>.
Hi Eleanore,
we understand side-input as something more general than simple broadcast
input, see FLIP-17 for details [1].
If a broadcast fits your use case, you can use that of course. We are
aiming for something, where a side input can also be co-partitioned. We are
currently laying the foundations for that feature.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API
On Thu, Feb 27, 2020 at 6:46 AM Jin Yi <el...@gmail.com> wrote:
> Hi All,
>
> there is a recent published article in the flink official website for
> running beam on top of flink
> https://flink.apache.org/ecosystem/2020/02/22/apache-beam-how-beam-runs-on-top-of-flink.html
>
> In the article:
>
> - You get additional features like side inputs and cross-language
> pipelines that are not supported natively in Flink but only supported when
> using Beam with Flink
>
> Ultimately, Beam pipeline will be translated into Flink job. So does
> beam's side input translates into Flink Broadcast stream?
>
> If I look at org.apache.beam.runners.flink.FlinkStreamingTransformTranslators,
> it looks like converting the the side input into broadcast stream, then why
> it says Flink does not support it natively?
>
> Thanks a lot!
> Eleanore
>
>