You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@beam.apache.org by JingsongLee <lz...@aliyun.com> on 2017/06/13 06:38:09 UTC

[DISCUSS] Bundle in Flink Runner

Hi everyone,
I take a discussion to the implement of real bundle in Flink Runner.
https://docs.google.com/document/d/1UzELM4nFu8SIeu-QJkbs0sv7Uzd1Ux4aXXM3cw4s7po/edit?usp=sharing
Feel free to comment/edit it.
Best, JingsongLee

Re: [DISCUSS] Bundle in Flink Runner

Posted by JingsongLee <lz...@aliyun.com>.

Thanks Aljoscha, 
I will add these related links to the docs.
Best, JingsongLee
------------------------------------------------------------------
From:Aljoscha Krettek <al...@apache.org>
Time:2017 Jun 14 (Wed) 20:56
To:dev <de...@beam.apache.org>; JingsongLee <lz...@aliyun.com>
Subject:Re: [DISCUSS] Bundle in Flink Runner
Hi,
Thanks for summarising everything that was discussed so far and also for coming up with a good implementation plan! The plan looks good, it will unblock proper bundle support on Beam because we cannot wait for Flink to support bundles the way we like it. 
I also want to quickly highlight these two issues because you mentioned bundle-processing and caching for GroupByKey/ReduceFnRunner as well: - https://issues.apache.org/jira/browse/BEAM-956: Execute ReduceFnRunner Directly in Flink Runner - https://issues.apache.org/jira/browse/BEAM-1850: Improve interplay between PushbackSideInputRunner and GroupAlsoByWindowViaWindowSetDoFn
They are not prerequisites for the bundling work but they are somewhat related.
Best,Aljoscha
 - 
On 13. Jun 2017, at 08:38, JingsongLee <lz...@aliyun.com> wrote:
Hi everyone,
I take a discussion to the implement of real bundle in Flink Runner.
https://docs.google.com/document/d/1UzELM4nFu8SIeu-QJkbs0sv7Uzd1Ux4aXXM3cw4s7po/edit?usp=sharing
Feel free to comment/edit it.
Best, JingsongLee

Re: [DISCUSS] Bundle in Flink Runner

Posted by Aljoscha Krettek <al...@apache.org>.

Hi,

Thanks for summarising everything that was discussed so far and also for coming up with a good implementation plan! The plan looks good, it will unblock proper bundle support on Beam because we cannot wait for Flink to support bundles the way we like it. 

I also want to quickly highlight these two issues because you mentioned bundle-processing and caching for GroupByKey/ReduceFnRunner as well:
 - https://issues.apache.org/jira/browse/BEAM-956: <https://issues.apache.org/jira/browse/BEAM-956:> Execute ReduceFnRunner Directly in Flink Runner
 - https://issues.apache.org/jira/browse/BEAM-1850: <https://issues.apache.org/jira/browse/BEAM-1850:> Improve interplay between PushbackSideInputRunner and GroupAlsoByWindowViaWindowSetDoFn

They are not prerequisites for the bundling work but they are somewhat related.

Best,
Aljoscha

 - 
> On 13. Jun 2017, at 08:38, JingsongLee <lz...@aliyun.com> wrote:
> 
> Hi everyone,
> I take a discussion to the implement of real bundle in Flink Runner.
> https://docs.google.com/document/d/1UzELM4nFu8SIeu-QJkbs0sv7Uzd1Ux4aXXM3cw4s7po/edit?usp=sharing
> Feel free to comment/edit it.
> Best, JingsongLee